Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.
Automate repetitive workflows and reduce manual effort
Example: Generate reports, summarize documents, draft communications
Save 3-5 hours per week on routine tasks
Learn new skills, understand complex topics, get expert guidance
Example: Explain concepts, provide examples, suggest learning resources
Accelerate learning and skill development by 2x
Enhance output quality through reviews, suggestions, and refinements
Example: Review drafts, suggest improvements, catch errors
Improve work quality by 30-40% with less effort
Time estimate: 15-45 minutes depending on use case complexity
Prerequisites
Submit your Claude Code skill and start earning
epx add nvidia/skills/nemo-mbridge-perf-hierarchical-context-parallelCompatible with
Supported agents
| name | nemo-mbridge-perf-hierarchical-context-parallel |
| description | Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification. |
| license | Apache-2.0 |
| when_to_use | Scaling context parallelism beyond KV heads, or investigating a commit that changed CP config and caused OOM or a regression; 'hierarchical_context_parallel_sizes', 'a2a+p2p', 'hierarchical CP', 'CP beyond KV heads', 'multi-level CP'. |
This skill covers hierarchical context parallelism: nested context-parallel process
groups used by cp_comm_type="a2a+p2p" and configured with
hierarchical_context_parallel_sizes.
For what hierarchical CP is, when to use it, and the decision tree
(a2a+p2p vs pure a2a vs p2p), see:
Minimal Bridge override:
cfg.model.context_parallel_size = 4
cfg.model.cp_comm_type = "a2a+p2p"
cfg.model.hierarchical_context_parallel_sizes = [2, 2]
cfg.dist.use_decentralized_pg = False
Required constraints:
prod(hierarchical_context_parallel_sizes) == context_parallel_sizeseq_length % (2 * context_parallel_size) == 0>= 1.12.0Upstream config and validation:
context_parallel_size: int = 1
"""Splits network input along sequence dimension across GPU ranks."""
hierarchical_context_parallel_sizes: Optional[list[int]] = None
"""Degrees of the hierarchical context parallelism. Users should provide a list to specify
the sizes for different levels. Taking the a2a+p2p cp comm type as example, it contains
groups of two levels, so the first value of the list indicates the group size of the a2a
communication type, and the second value indicates the group size of the p2p communication
type.
"""
if args.hierarchical_context_parallel_sizes:
from numpy import prod
assert args.context_parallel_size == prod(args.hierarchical_context_parallel_sizes)
if "a2a+p2p" in args.cp_comm_type:
assert args.hierarchical_context_parallel_sizes is not None, \
"--hierarchical-context-parallel-sizes must be set when a2a+p2p is used in cp comm"
Bridge MPU path:
parallel_state.initialize_model_parallel(
...
context_parallel_size=model_config.context_parallel_size,
hierarchical_context_parallel_sizes=model_config.hierarchical_context_parallel_sizes,
...
)
...
return ProcessGroupCollection.use_mpu_process_groups()
Bridge decentralized-PG path:
pg_collection = ProcessGroupCollection(
...
cp=cp_pg,
tp_cp=tp_cp_pg,
hcp=None,
ep=ep_pg,
...
)
The code anchors above show the config declarations and argument validation.
TransformerConfig.__post_init__ enforces that a2a+p2p requires HCP sizes and the product matches CP.
parallel_state.initialize_model_parallel creates hierarchical CP sub-groups
when HCP sizes are provided via create_hierarchical_groups. Bridge currently
gets those groups through the MPU-backed ProcessGroupCollection.
TEDotProductAttention passes the hierarchical groups to Transformer Engine
when a2a+p2p is used. Requires Transformer Engine >= 1.12.0.
use_decentralized_pg=True, Bridge initializes flat CP groups and leaves HCP unset.hierarchical_context_parallel_sizes.a2a+p2p without setting hierarchical_context_parallel_sizes, MCore now asserts. Older versions would silently disable CP communication, so each rank attended only to its local chunk and produced artificially high throughput with broken gradients.prod(hierarchical_context_parallel_sizes) must exactly equal context_parallel_size. A mismatch triggers an assertion.HIERARCHICAL_CONTEXT_PARALLEL_GROUPS being created. If you only see CONTEXT_PARALLEL_GROUP, HCP is not active.No dedicated Bridge end-to-end test exists yet for HCP (see @skills/nemo-mbridge-perf-hierarchical-context-parallel/card.yaml
follow_up_validation). Use the existing unit tests and log inspection instead.
Run the decentralized-PG unit test to confirm the flat-CP behavior is preserved:
uv run python -m pytest tests/unit_tests/training/test_decentralized_pg.py -q
For a manual smoke check, launch a 4-GPU run with a small recipe and
cp_comm_type=a2a+p2p plus hierarchical_context_parallel_sizes=[2,2]:
CUDA_VISIBLE_DEVICES=0,1,2,3 uv run python -m torch.distributed.run --nproc_per_node=4 \
scripts/training/run_recipe.py \
--recipe llama32_1b_pretrain_config \
model.context_parallel_size=4 \
model.cp_comm_type=a2a+p2p \
"model.hierarchical_context_parallel_sizes=[2,2]" \
train.train_iters=2
Success criteria:
HIERARCHICAL_CONTEXT_PARALLEL_GROUPS being createdCONTEXT_PARALLEL_GROUP, HCP is not activeAI-first code editor with Composer
Before installing skills in Cursor, ensure your development environment meets these requirements:
node --versionnemo-mbridge-perf-hierarchical-context-parallelExecute the skills CLI command in your project's root directory to begin installation:
epx add nvidia/skills/nemo-mbridge-perf-hierarchical-context-parallelFetches nemo-mbridge-perf-hierarchical-context-parallel from nvidia/skills and configures it for Cursor.
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Confirm successful installation by checking the skill directory location:
.cursor/skills/nemo-mbridge-perf-hierarchical-context-parallel
Restart Cursor to activate nemo-mbridge-perf-hierarchical-context-parallel. Access via /nemo-mbridge-perf-hierarchical-context-parallel in your agent's command palette.
Learn more:View source on GitHubSkills CLI docsAbout CursorWhat are agent skills?
Steps
Common Pitfalls
✓ Do
✗ Don't
💡 Pro Tips
✓ Use when
Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.
✗ Avoid when
Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.
nemo-mbridge-perf-megatron-fsdp
nvidia/skills · nemo-mbridge · same repo
nemo-mbridge-perf-cpu-offloading
nvidia/skills · nemo-mbridge · same repo
earth2studio-install
nvidia/skills · earth2studio · same repo
hsb-test
nvidia/skills · hsb · same repo
jetson-init-image
nvidia/skills · jetson · same repo
cuopt-server-common
nvidia/skills · cuopt · same repo
nemo-mbridge-perf-hierarchical-context-parallel is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
nemo-mbridge-perf-hierarchical-context-parallel fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
We added nemo-mbridge-perf-hierarchical-context-parallel from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
nemo-mbridge-perf-hierarchical-context-parallel reduced setup friction for our internal harness; good balance of opinion and flexibility.
nemo-mbridge-perf-hierarchical-context-parallel fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
nemo-mbridge-perf-hierarchical-context-parallel has been reliable in day-to-day use. Documentation quality is above average for community skills.
Solid pick for teams standardizing on skills: nemo-mbridge-perf-hierarchical-context-parallel is focused, and the summary matches what you get after install.
Keeps context tight: nemo-mbridge-perf-hierarchical-context-parallel is the kind of skill you can hand to a new teammate without a long onboarding doc.
We added nemo-mbridge-perf-hierarchical-context-parallel from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
Solid pick for teams standardizing on skills: nemo-mbridge-perf-hierarchical-context-parallel is focused, and the summary matches what you get after install.
showing 1-10 of 49