(https://github.githubassets.com) (https://avatars.githubusercontent.com) (https://github-cloud.s3.amazonaws.com) (https://user-images.githubusercontent.com/) (https://github.githubassets.com) (https://avatars.githubusercontent.com)   (https://github.githubassets.com/assets/react-e27d1b3e03961e68.js)  (https://github.githubassets.com/assets/react-compiler-runtime-4610bd6d3de9c049.js)  (https://github.githubassets.com/assets/scheduler-58b860b049ca307c.js)  (https://github.githubassets.com/assets/react-dom-e5fd46a22d5c4058.js)  (https://github.githubassets.com/assets/react-dom-client-1b4a3ee065998cea.js)  (https://github.githubassets.com/assets/react-is-e0b593954b4706d8.js)  (https://github.githubassets.com/assets/react-jsx-runtime-4915cb0f5b3aff04.js)  (https://github.githubassets.com/assets/react-reconciler-8e99e505c4429605.js)  (../../github.githubassets.com/assets/light-3a3d2c708970e4e8.css)  (../../github.githubassets.com/assets/light_high_contrast-9c7b506bdc5ea73f.css)  (../../github.githubassets.com/assets/dark-c9dacb4ec19616db.css)  (../../github.githubassets.com/assets/dark_high_contrast-5ef38214a3eabc7d.css)                (../../github.githubassets.com/assets/primer-primitives-02e066689466a8cd.css)  (../../github.githubassets.com/assets/primer-38ae573fa780d76a.css)  (../../github.githubassets.com/assets/global-58d1122073b19732.css)  (../../github.githubassets.com/assets/github-0b4a3ee3d18dc2f0.css)  (../../github.githubassets.com/assets/repository-542ca59b4b92999d.css)  (../../github.githubassets.com/assets/code-b25a4886e30e3e5f.css)  (https://github.githubassets.com/assets/wp-runtime-333065c27df0244b.js)  (https://github.githubassets.com/assets/app-foundation-799625033ee8341e.js)  (https://github.githubassets.com/assets/app-runtime-75fe7337a5ae1f43.js)  (https://github.githubassets.com/assets/fetch-utilities-6eb475aaa076ff11.js)  (https://github.githubassets.com/assets/ser-1ce9ba53965ee000.js)  (https://github.githubassets.com/assets/tp-e043b09789480cb5.js)  (../../github.githubassets.com/assets/app-runtime.11d24f4ef5060b0f.module.css)  (https://github.githubassets.com/assets/catalyst-5bc36fa5a846f5a0.js)  (https://github.githubassets.com/assets/selector-observer-7ae05416e4a383ae.js)  (https://github.githubassets.com/assets/relative-time-element-b2d8b7dede692c9f.js)  (https://github.githubassets.com/assets/by-473024a3f6cb3892.js)  (https://github.githubassets.com/assets/ja9-b94de714266ba389.js)  (https://github.githubassets.com/assets/3h2-5f8899065b81d778.js)  (https://github.githubassets.com/assets/t2-19741658de225445.js)  (https://github.githubassets.com/assets/z8-5c899bb34cf0fc51.js)  (https://github.githubassets.com/assets/hj-ece3bec966e8a2b6.js)  (https://github.githubassets.com/assets/wc-33edcf25ba11d33f.js)  (https://github.githubassets.com/assets/runtime-helpers-b7e0a61126a143c7.js)  (https://github.githubassets.com/assets/aria-live-edd851c704c6b3ca.js)  (https://github.githubassets.com/assets/hotkey-a82e6cb07443fd1c.js)  (https://github.githubassets.com/assets/react-core-7fe869c9c8566fc0.js)  (https://github.githubassets.com/assets/a96-f44dcca0945fa24d.js)  (https://github.githubassets.com/assets/ur-b820e79826e6a550.js)  (https://github.githubassets.com/assets/4t-f1181f796eb9ea15.js)  (https://github.githubassets.com/assets/pb0-b19b273276c337a2.js)  (https://github.githubassets.com/assets/6n-b72a71e2477d5d76.js)  (https://github.githubassets.com/assets/z-b2cf9f86cb4bed3b.js)  (https://github.githubassets.com/assets/9k-707f3c96c4acaae5.js)  (https://github.githubassets.com/assets/0r-4c4af04176d5b5a2.js)  (https://github.githubassets.com/assets/ow-98d10140e8228c74.js)  (https://github.githubassets.com/assets/b3-8c6a9e264988c148.js)  (https://github.githubassets.com/assets/eonf-7297a0ec85fc5281.js)  (https://github.githubassets.com/assets/od3-5d7c03317ee3745b.js)  (https://github.githubassets.com/assets/yga-dd50cf95593467fe.js)  (../../github.githubassets.com/assets/react-core.fab33199790f05d6.module.css)  (https://github.githubassets.com/assets/n9c-3506ddcf805d5421.js)  (https://github.githubassets.com/assets/li-902486fa635be114.js)  (https://github.githubassets.com/assets/primer-react-aa472ad9b7027c95.js)  (https://github.githubassets.com/assets/op-76db8804e205b5d5.js)  (https://github.githubassets.com/assets/nc-2485e88d8a01d9c2.js)  (https://github.githubassets.com/assets/qmp-308230626a2aedb0.js)  (https://github.githubassets.com/assets/j6-f38514fb943db6ec.js)  (https://github.githubassets.com/assets/t7l-699ca510ac327acf.js)  (https://github.githubassets.com/assets/4u-bf6e08201764cb0a.js)  (https://github.githubassets.com/assets/s0-307dd28b274d0fad.js)  (https://github.githubassets.com/assets/y5-4fcf85a93b5b06b0.js)  (https://github.githubassets.com/assets/zd-61606dd1c14634aa.js)  (https://github.githubassets.com/assets/7v-88e5da497f84e96d.js)  (https://github.githubassets.com/assets/15-f26349519bec5482.js)  (https://github.githubassets.com/assets/5w-452c3a937483ff49.js)  (https://github.githubassets.com/assets/la-27ea91f01fa7ffcf.js)  (https://github.githubassets.com/assets/a3-535693ca876f3978.js)  (https://github.githubassets.com/assets/fx0-05f68952cd4c4b54.js)  (https://github.githubassets.com/assets/tjn-c8192b2dabd1ae8c.js)  (https://github.githubassets.com/assets/owu-d27475511623d054.js)  (https://github.githubassets.com/assets/j8z-2b6baec9227eb98c.js)  (https://github.githubassets.com/assets/znp-671ea2c74aed7e30.js)  (https://github.githubassets.com/assets/3s-ebbc335534e40b2a.js)  (https://github.githubassets.com/assets/gj-b68aa2e46dac0e69.js)  (https://github.githubassets.com/assets/4y-9b20c7f1b1e68334.js)  (https://github.githubassets.com/assets/0-8bb1b20ebce778dd.js)  (https://github.githubassets.com/assets/jn-8d682b11fce7d25e.js)  (https://github.githubassets.com/assets/fn-80ae055b0fa5b680.js)  (https://github.githubassets.com/assets/q6j-3e0ae2c7fe8108f2.js)  (https://github.githubassets.com/assets/b2-35e229a50d0f3d05.js)  (https://github.githubassets.com/assets/pgx-dd79b7c9d5ccc946.js)  (https://github.githubassets.com/assets/lrk-ea49e115e48ba34b.js)  (https://github.githubassets.com/assets/mlz-b5402a5997108f93.js)  (https://github.githubassets.com/assets/dx-4c2b444c4f196d9b.js)  (https://github.githubassets.com/assets/sm-22346830d5e5a763.js)  (https://github.githubassets.com/assets/i2-5e26ac37ee961d06.js)  (https://github.githubassets.com/assets/zxp-5a6b6d5f0d4b8a6a.js)  (https://github.githubassets.com/assets/43-efe637f521d688f0.js)  (https://github.githubassets.com/assets/pr-e6c8a171cb829011.js)  (https://github.githubassets.com/assets/jh-d137d2245bb68bfc.js)  (https://github.githubassets.com/assets/0yx-6e44b5cba2d0d1f7.js)  (https://github.githubassets.com/assets/k54-6cf64d936e253d31.js)  (https://github.githubassets.com/assets/up-9ba623fe31b3ad8b.js)  (https://github.githubassets.com/assets/gd-c0d7e6321e1379df.js)  (https://github.githubassets.com/assets/d0-05f8d14a7470eb9e.js)  (https://github.githubassets.com/assets/c28-7bbf3d68f4dce8b5.js)  (https://github.githubassets.com/assets/ti-5a1f71482f5e6a07.js)  (https://github.githubassets.com/assets/y2-2f581d0564f88f89.js)  (https://github.githubassets.com/assets/9xk-7f6912c4c8ddb34a.js)  (https://github.githubassets.com/assets/6z-8b64f176b6154ae7.js)  (../../github.githubassets.com/assets/primer-react-css.41e8289c1ff152bd.module.css)  (../../github.githubassets.com/assets/0yx.87256cbd6216cf2c.module.css)  (../../github.githubassets.com/assets/k54.e9cc8c0ec9e8e133.module.css)  (../../github.githubassets.com/assets/code-view.f4c02ea21c64248d.module.css)  GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models · GitHub      (repo_source) (https://github.githubassets.com/)   (https://github.com/opensearch.xml) (GitHub) (https://github.com/fluidicon.png) (GitHub)                        (https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg) (https://github.githubassets.com/favicons/favicon.png) (../../github.githubassets.com/favicons/favicon.svg)  (../manifest.json)   (DeepInfer.html#start-of-content) Skip to content   (https://github.githubassets.com/assets/bbj-120f614daab2ee1d.js)  (https://github.githubassets.com/assets/keyboard-shortcuts-dialog-1bdeac3a649d34a9.js)  (../../github.githubassets.com/assets/primer-react-css.41e8289c1ff152bd.module.css)  (../../github.githubassets.com/assets/keyboard-shortcuts-dialog.1b0e6d5f3384752f.module.css)    (https://github.githubassets.com/assets/g6m-a05cbcd94ac196d8.js)  (https://github.githubassets.com/assets/app-install-banner-partial-782af9a8839d9cd1.js)  (../../github.githubassets.com/assets/primer-react-css.41e8289c1ff152bd.module.css)  (../../github.githubassets.com/assets/app-install-banner-partial.f54d7f77f770782d.module.css)    (https://github.githubassets.com/assets/et9-6f05589c8621e03c.js)  (https://github.githubassets.com/assets/id-6e69bc70bac6b188.js)  (https://github.githubassets.com/assets/fz-cb55e83c29d0a351.js)  (https://github.githubassets.com/assets/jbz-52a745dad9e9a69a.js)  (https://github.githubassets.com/assets/h4-1b8e14009663edc9.js)  (https://github.githubassets.com/assets/mhh-8ec07c1858d988b9.js)  (https://github.githubassets.com/assets/97-0c9e3d8ca57bb36c.js)  (https://github.githubassets.com/assets/marketing-header-71cfbf6579799ab6.js)  (../../github.githubassets.com/assets/primer-react-css.41e8289c1ff152bd.module.css)  (../../github.githubassets.com/assets/primer-react-brand-css.011e078829877390.module.css)  (../../github.githubassets.com/assets/marketing-header.8e5d225c1277248f.module.css)  Navigation Menu       (https://github.com/)    (https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdeepseek-ai%2FDeepInfer) Sign in       Appearance settings   Platform   AI CODE CREATION (https://github.com/features/copilot)    GitHub Copilot Write better code with AI    (https://github.com/features/ai/github-app)   GitHub Copilot app Direct agents from issue to merge    (https://github.com/mcp)    MCP Registry Integrate external tools       DEVELOPER WORKFLOWS (https://github.com/features/actions)   Actions Automate any workflow    (https://github.com/features/codespaces)    Codespaces Instant dev environments    (https://github.com/features/issues)    Issues Plan and track work    (https://github.com/features/code-review)   Code Review Manage code changes    (https://github.com/features/code-quality)    Code Quality Enforce quality at merge       APPLICATION SECURITY (https://github.com/security/advanced-security)   GitHub Advanced Security Find and fix vulnerabilities    (https://github.com/security/advanced-security/code-security)   Code security Secure your code as you build    (https://github.com/security/advanced-security/secret-protection)   Secret protection Stop leaks before they start       EXPLORE (https://github.com/why-github) Why GitHub   (https://docs.github.com) Documentation     (https://github.blog) Blog     (https://github.blog/changelog) Changelog     (https://github.com/marketplace) Marketplace       (https://github.com/features) View all features         Solutions   BY COMPANY SIZE (https://github.com/enterprise) Enterprises   (https://github.com/team) Small and medium teams   (https://github.com/enterprise/startups) Startups   (https://github.com/solutions/industry/nonprofits) Nonprofits      BY USE CASE (https://github.com/solutions/use-case/app-modernization) App Modernization   (https://github.com/solutions/use-case/devsecops) DevSecOps   (https://github.com/solutions/use-case/devops) DevOps   (https://github.com/solutions/use-case/ci-cd) CI/CD   (https://github.com/solutions/use-case) View all use cases         BY INDUSTRY (https://github.com/solutions/industry/healthcare) Healthcare   (https://github.com/solutions/industry/financial-services) Financial services   (https://github.com/solutions/industry/manufacturing) Manufacturing   (https://github.com/solutions/industry/government) Government   (https://github.com/solutions/industry) View all industries          (https://github.com/solutions) View all solutions         Resources   EXPLORE BY TOPIC (https://github.com/resources/articles?topic=ai) AI   (https://github.com/resources/articles?topic=software-development) Software Development   (https://github.com/resources/articles?topic=devops) DevOps   (https://github.com/resources/articles?topic=security) Security   (https://github.com/resources/articles) View all topics         EXPLORE BY TYPE (https://github.com/customer-stories) Customer stories   (https://github.com/resources/events) Events & webinars   (https://github.com/resources/whitepapers) Ebooks & reports   (https://github.com/solutions/executive-insights) Business insights   (https://skills.github.com) GitHub Skills        SUPPORT & SERVICES (https://docs.github.com) Documentation     (https://support.github.com) Customer support     (https://github.com/orgs/community/discussions) Community forum   (https://github.com/trust-center) Trust center   (https://github.com/partners) Partners       (https://github.com/resources) View all resources         Open Source   COMMUNITY (https://github.com/open-source/sponsors)   GitHub Sponsors Fund open source developers       PROGRAMS (https://securitylab.github.com) Security Lab     (https://maintainers.github.com) Maintainer Community     (https://stars.github.com) GitHub Stars     (https://archiveprogram.github.com) Archive Program        REPOSITORIES (https://github.com/topics) Topics   (https://github.com/trending) Trending   (https://github.com/collections) Collections          Enterprise   ENTERPRISE SOLUTIONS (https://github.com/enterprise)   Enterprise platform AI-powered developer platform       AVAILABLE ADD-ONS (https://github.com/security/advanced-security)   GitHub Advanced Security Enterprise-grade security features    (https://github.com/features/copilot/copilot-business)    Copilot for Business Enterprise-grade AI features    (https://github.com/enterprise/premium-support)   Premium Support Enterprise-grade 24/7 support           (https://github.com/pricing) Pricing      Search /       (https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdeepseek-ai%2FDeepInfer) Sign in    (https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=deepseek-ai%2FDeepInfer) Sign up       Appearance settings            You signed in with another tab or window. (DeepInfer.html) Reload to refresh your session. You signed out in another tab or window. (DeepInfer.html) Reload to refresh your session. You switched accounts on another tab or window. (DeepInfer.html) Reload to refresh your session.    Dismiss alert     (https://github.githubassets.com/assets/r7n-381a130887b7aeb7.js)  (https://github.githubassets.com/assets/gex-d368cfd87cd0aa60.js)  (https://github.githubassets.com/assets/t5-60a7d058a0d3af98.js)  (https://github.githubassets.com/assets/5l-a09e91e64044f066.js)  (https://github.githubassets.com/assets/1al-dbc76bfa747870da.js)  (https://github.githubassets.com/assets/9t-064ddb7b929b34ad.js)  (https://github.githubassets.com/assets/xpm-699cadc1a20b7325.js)  (https://github.githubassets.com/assets/24-8bc4d151664bf03d.js)  (https://github.githubassets.com/assets/4h-9938b919ef32394f.js)  (https://github.githubassets.com/assets/3j7-c3f75949a1cb5d34.js)  (https://github.githubassets.com/assets/aj-6d3aa2e6c4939409.js)  (https://github.githubassets.com/assets/wyn-832e1b37386e4deb.js)  (https://github.githubassets.com/assets/zc-ed0a4d5a713e09b5.js)  (https://github.githubassets.com/assets/li6-9946ed6b82f92046.js)  (https://github.githubassets.com/assets/wv-0c5700130d2a015b.js)  (https://github.githubassets.com/assets/global-nav-bar-4e828fe834782607.js)  (../../github.githubassets.com/assets/primer-react-css.41e8289c1ff152bd.module.css)  (../../github.githubassets.com/assets/0yx.87256cbd6216cf2c.module.css)  (../../github.githubassets.com/assets/k54.e9cc8c0ec9e8e133.module.css)  (../../github.githubassets.com/assets/global-nav-bar.b5e9f0aa7cd75729.module.css)  (https://github.com/deepseek-ai) deepseek-ai   (DeepInfer.html) DeepInfer        Repository navigation  (DeepInfer.html)    Code   (https://github.com/deepseek-ai/DeepInfer/pulls)    Pull requests 3 (3)    (https://github.com/deepseek-ai/DeepInfer/actions)    Actions   (https://github.com/deepseek-ai/DeepInfer/projects)    Projects   (https://github.com/deepseek-ai/DeepInfer/security)    Security and quality   (https://github.com/deepseek-ai/DeepInfer/pulse)    Insights     Moreitems              (../../avatars.githubusercontent.com/u/148330874@s=60&v=4)   (DeepInfer.html)   (deepseek-ai)  (DeepInfer) (DeepInfer.html) DeepInfer   Public    (https://github.com/login?return_to=%2Fdeepseek-ai%2FDeepInfer)    Notifications   You must be signed in to change notification settings  (https://github.com/login?return_to=%2Fdeepseek-ai%2FDeepInfer)    Fork253  (253 )     (https://github.com/login?return_to=%2Fdeepseek-ai%2FDeepInfer)    Star3.5k  (3.5k )    You must be signed in to star a repository      About  DeepInfer: High-performance, large-scale inference framework for DeepSeek models Resources  (DeepInfer.html#readme-ov-file)   Readme   (DeepInfer.html#Apache-2.0-1-ov-file)   Apache-2.0 license   Contributing  (DeepInfer.html#contributing-ov-file)   Contributing   (https://github.com/deepseek-ai/DeepInfer/activity)   Activity   (https://github.com/deepseek-ai/DeepInfer/custom-properties)   Custom properties    Stars    3.5k stars  Watchers    2 watching  Forks  (https://github.com/deepseek-ai/DeepInfer/forks)   253  forks     (https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fdeepseek-ai%2FDeepInfer&report=deepseek-ai+%28user%29) Report repository            main           (https://github.com/deepseek-ai/DeepInfer/branches)    Branches   (https://github.com/deepseek-ai/DeepInfer/tags)    Tags    (https://github.com/deepseek-ai/DeepInfer/branches)    (https://github.com/deepseek-ai/DeepInfer/tags)          (Go to file) ()      Go to file        Code         Open more actions menu     Latest commit   History (https://github.com/deepseek-ai/DeepInfer/commits/main/)    2 Commits     (https://github.com/deepseek-ai/DeepInfer/commits/main/)      2 Commits      Folders and files Name  Name  (Last commit message) Last commit message   (Last commit date) Last commit date       (csrc) (https://github.com/deepseek-ai/DeepInfer/tree/main/csrc) csrc        (csrc) (https://github.com/deepseek-ai/DeepInfer/tree/main/csrc) csrc              (deep_cluster) (https://github.com/deepseek-ai/DeepInfer/tree/main/deep_cluster) deep_cluster        (deep_cluster) (https://github.com/deepseek-ai/DeepInfer/tree/main/deep_cluster) deep_cluster              (deepinfer) (https://github.com/deepseek-ai/DeepInfer/tree/main/deepinfer) deepinfer        (deepinfer) (https://github.com/deepseek-ai/DeepInfer/tree/main/deepinfer) deepinfer              (doc) (https://github.com/deepseek-ai/DeepInfer/tree/main/doc) doc        (doc) (https://github.com/deepseek-ai/DeepInfer/tree/main/doc) doc              (eval) (https://github.com/deepseek-ai/DeepInfer/tree/main/eval) eval        (eval) (https://github.com/deepseek-ai/DeepInfer/tree/main/eval) eval              (examples) (https://github.com/deepseek-ai/DeepInfer/tree/main/examples) examples        (examples) (https://github.com/deepseek-ai/DeepInfer/tree/main/examples) examples              (scripts) (https://github.com/deepseek-ai/DeepInfer/tree/main/scripts) scripts        (scripts) (https://github.com/deepseek-ai/DeepInfer/tree/main/scripts) scripts              (tests) (https://github.com/deepseek-ai/DeepInfer/tree/main/tests) tests        (tests) (https://github.com/deepseek-ai/DeepInfer/tree/main/tests) tests              (vendor) (https://github.com/deepseek-ai/DeepInfer/tree/main/vendor) vendor        (vendor) (https://github.com/deepseek-ai/DeepInfer/tree/main/vendor) vendor              (.clang-format) (https://github.com/deepseek-ai/DeepInfer/blob/main/.clang-format) .clang-format        (.clang-format) (https://github.com/deepseek-ai/DeepInfer/blob/main/.clang-format) .clang-format              (.gitignore) (https://github.com/deepseek-ai/DeepInfer/blob/main/.gitignore) .gitignore        (.gitignore) (https://github.com/deepseek-ai/DeepInfer/blob/main/.gitignore) .gitignore              (.gitmodules) (https://github.com/deepseek-ai/DeepInfer/blob/main/.gitmodules) .gitmodules        (.gitmodules) (https://github.com/deepseek-ai/DeepInfer/blob/main/.gitmodules) .gitmodules              (CONTRIBUTING.md) (https://github.com/deepseek-ai/DeepInfer/blob/main/CONTRIBUTING.md) CONTRIBUTING.md        (CONTRIBUTING.md) (https://github.com/deepseek-ai/DeepInfer/blob/main/CONTRIBUTING.md) CONTRIBUTING.md              (LICENSE) (https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE) LICENSE        (LICENSE) (https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE) LICENSE              (Makefile) (https://github.com/deepseek-ai/DeepInfer/blob/main/Makefile) Makefile        (Makefile) (https://github.com/deepseek-ai/DeepInfer/blob/main/Makefile) Makefile              (README.md) (https://github.com/deepseek-ai/DeepInfer/blob/main/README.md) README.md        (README.md) (https://github.com/deepseek-ai/DeepInfer/blob/main/README.md) README.md              (build.sh) (https://github.com/deepseek-ai/DeepInfer/blob/main/build.sh) build.sh        (build.sh) (https://github.com/deepseek-ai/DeepInfer/blob/main/build.sh) build.sh              (pyproject.toml) (https://github.com/deepseek-ai/DeepInfer/blob/main/pyproject.toml) pyproject.toml        (pyproject.toml) (https://github.com/deepseek-ai/DeepInfer/blob/main/pyproject.toml) pyproject.toml              (requirements-ascend.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-ascend.txt) requirements-ascend.txt        (requirements-ascend.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-ascend.txt) requirements-ascend.txt              (requirements-dev.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-dev.txt) requirements-dev.txt        (requirements-dev.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-dev.txt) requirements-dev.txt              (requirements-nvidia.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-nvidia.txt) requirements-nvidia.txt        (requirements-nvidia.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements-nvidia.txt) requirements-nvidia.txt              (requirements.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements.txt) requirements.txt        (requirements.txt) (https://github.com/deepseek-ai/DeepInfer/blob/main/requirements.txt) requirements.txt              (setup.py) (https://github.com/deepseek-ai/DeepInfer/blob/main/setup.py) setup.py        (setup.py) (https://github.com/deepseek-ai/DeepInfer/blob/main/setup.py) setup.py            View all files        Repository files navigation  (DeepInfer.html#)    README   (DeepInfer.html#)    Contributing   (DeepInfer.html#)    Apache-2.0 license     Moreitems               DeepInfer (DeepInfer.html#deepinfer)     DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale. Supported models: DeepSeek-V4.1-Flash  Supported features: Optimized kernels, model and sampler runtime Parallelism: Attention DP / balanced CP, MoE EP + EPLB DSpark and confidence-scheduled verification SWA bounded replay Asynchronous scheduling (zero-overhead batch scheduler) Batch-invariant inference Deterministic sampling Structured outputs  Feature details and configuration Optimized kernels: (https://github.com/deepseek-ai/FlashMLA) FlashMLA , (https://github.com/deepseek-ai/DeepGEMM) DeepGEMM , (https://github.com/deepseek-ai/DeepEP) DeepEP , (https://github.com/deepseek-ai/TileKernels) TileKernels , (https://github.com/deepseek-ai/DeepSelect) DeepSelect , and custom (https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model_executor/layers/tilelang_ops) TileLang kernels . DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1 ; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5). Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length. Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default. Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only). Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only). Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams ; see (https://github.com/deepseek-ai/DeepInfer/blob/main/examples/_structured_output.py) examples .   Supported hardware: NVIDIA SM100 GPUs Huawei Ascend 950 NPUs  Required for a complete inference service (not included in this release): A KV cache system VE/Prefill/Decode disaggregation (most required engine functionality is already implemented) Load balancing across data-parallel replicas for both prefill and decode  Getting started (DeepInfer.html#getting-started)     Prerequisites (DeepInfer.html#prerequisites)     Linux GCC >= 13.0 Python 3.12 CUDA 13 (NVIDIA only) CANN 9.2 (Ascend only)  Build from source (DeepInfer.html#build-from-source)     Before building DeepInfer, please create and activate an empty Python 3.12 environment. # Use `ascend` instead for the Ascend platform # Keep it exported in the environment where you run DeepInfer export DEEPINFER_ACCELERATOR=nvidia
./build.sh  Offline inference (DeepInfer.html#offline-inference)     The default checkpoint, (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017) deepseek-ai/DeepSeek-V4.1-Flash-1017  , is
downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to
run from a local copy instead. python examples/offline_inference.py  See (https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md) examples for details. Performance benchmarks (DeepInfer.html#performance-benchmarks)     Prefill Decode   (https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/prefill_pareto.png) (Prefill: non-cached input throughput vs. latency)   (https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/decode_pareto.png) (Decode: output throughput vs. per-user speed)       Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see (https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/run_sweep.sh) run_sweep.sh . Evaluation benchmarks (DeepInfer.html#evaluation-benchmarks)     See (https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md) eval/README.md . License (DeepInfer.html#license)     DeepInfer is released under the Apache License 2.0. See (https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE) LICENSE for details. Acknowledgments (DeepInfer.html#acknowledgments)     DeepInfer was originally built primarily on (https://github.com/vllm-project/vllm) vLLM , with additional inspiration from (https://github.com/sgl-project/sglang) SGLang and (https://github.com/NVIDIA/TensorRT-LLM) TensorRT-LLM . We thank their developers. Citation (DeepInfer.html#citation)     @misc {deepinfer2026 , title = { DeepInfer: High-performance, large-scale inference framework for DeepSeek models}  , author = { DeepSeek-AI}  , year = { 2026}  , publisher = { GitHub}  , howpublished = { \url{https://github.com/deepseek-ai/DeepInfer}}  ,
}           About  DeepInfer: High-performance, large-scale inference framework for DeepSeek models Resources  (DeepInfer.html#readme-ov-file)   Readme   (DeepInfer.html#Apache-2.0-1-ov-file)   Apache-2.0 license   Contributing  (DeepInfer.html#contributing-ov-file)   Contributing   (https://github.com/deepseek-ai/DeepInfer/activity)   Activity   (https://github.com/deepseek-ai/DeepInfer/custom-properties)   Custom properties    Stars    3.5k stars  Watchers    2 watching  Forks  (https://github.com/deepseek-ai/DeepInfer/forks)   253  forks     (https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fdeepseek-ai%2FDeepInfer&report=deepseek-ai+%28user%29) Report repository    Releases     Packages    Used by    Contributors    Languages                            Footer (https://github.com)    © 2026 GitHub, Inc.   Footer navigation (https://docs.github.com/site-policy/github-terms/github-terms-of-service) Terms  (https://docs.github.com/site-policy/privacy-policies/github-privacy-statement) Privacy  (https://github.com/security) Security  (https://www.githubstatus.com/) Status  (https://github.community/) Community  (https://docs.github.com/) Docs  (https://support.github.com?tags=dotcom-footer) Contact  Manage cookies    Do not share my personal information              You can’t perform that action at this time.