top of page

latest stuff in ai, directly in your inbox. 🤗

Thanks for submitting!

LongNet: Pioneering the Future of AI with a Leap to 1 Billion Token Transformers

Welcome to the era of large language models where scaling sequence length is not just a possibility, but an imperative demand. The race to tackle computational complexity and enhance model expressivity has been a defining challenge for developers worldwide. Today, we're thrilled to introduce a groundbreaking model that breaks free from these barriers - LongNet. This Transformer variant can scale sequence length to a staggering 1 billion tokens, without compromising on the performance of shorter sequences.

Soaring Sequence Lengths with Dilated Attention

The cornerstone of LongNet is the unique concept of 'dilated attention'. As the distance grows, dilated attention exponentially expands the attentive field, allowing the model to handle longer sequences. Dilated attention's ability to increase the model's awareness of distant tokens sets it apart from conventional models and makes billion-token Transformers a reality.

Key Advantages of LongNet

LongNet brings a host of benefits to the table. For starters, it demonstrates a linear computational complexity, a remarkable feat considering the scale it operates at. The model also showcases a logarithmic dependency between tokens, significantly reducing the computational burden.

LongNet can be employed as a distributed trainer for extremely long sequences, opening up new avenues in the realm of deep learning. What's more, the model's dilated attention serves as a drop-in replacement for standard attention and can be seamlessly integrated with existing Transformer-based optimizations.

Performance Metrics and Potential Applications

LongNet's experimental results reveal its exceptional capability, yielding strong performance in both long-sequence modeling and general language tasks. The model's versatility coupled with its robust scalability makes it a promising tool for handling extensive sequences.

With LongNet, it's possible to consider a whole corpus or even the entire Internet as a sequence. Imagine the potential: from profound text analytics on entire libraries of books to comprehensive web analysis for understanding global trends. The possibilities are truly endless.


LongNet signals a paradigm shift in the world of large language models. By scaling Transformers to 1 billion tokens, it offers unparalleled potential in processing and understanding extensive sequences. As we move forward, the introduction of LongNet will not just open up new possibilities, but could very well set a new standard in the field. The world stands on the brink of a revolutionary change in sequence modeling - and LongNet is leading the charge.

28 views0 comments



Snapy allows you to edit your videos with the power of ai. Save at least 30 minutes of editing time for a typical 5-10 minute long video.

- Trim silent parts of your videos
- Make your content more interesting for your audience
- Focus on making more quality content, we will take care of the editing

Landing AI

A platform to create and deploy custom computer vision projects.


An image enhancement platform.


A tool for face-morphing and memes.


SuperAGI is an open-source platform providing infrastructure to build autonomous AI agents.


A tool to create personalized fitness plans.


A tool to summarize lectures and educational materials.


A platform for emails productivity.


An all-in-one social media management tool.


A tool to generate personalized content.

Addy AI

A Google Chrome Exntesion as an email assistant.


A telegrambot to organize notes in Notion.

bottom of page