(https://gmpg.org/xfn/11) The systems that no one will test | Terra Incognita     (index.html)   (https://www.googletagmanager.com/)  (Terra Incognita » Feed) (https://blog.christianperone.com/feed/)  (Terra Incognita » Comments Feed) (https://blog.christianperone.com/comments/feed/)  (Terra Incognita » The systems that no one will test Comments Feed) (https://blog.christianperone.com/2026/09/the-systems-that-no-one-will-test/feed/)  (oEmbed (JSON)) (https://blog.christianperone.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fblog.christianperone.com%2F2026%2F09%2Fthe-systems-that-no-one-will-test%2F)  (oEmbed (XML)) (https://blog.christianperone.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fblog.christianperone.com%2F2026%2F09%2Fthe-systems-that-no-one-will-test%2F&format=xml)  (../../../wp-content/plugins/contact-form-7/includes/css/styles.css@ver=6.1.7.css)  (../../../wp-content/plugins/cookie-law-info/legacy/public/css/cookie-law-info-public.css@ver=3.5.6.css)  (../../../wp-content/plugins/cookie-law-info/legacy/public/css/cookie-law-info-gdpr.css@ver=3.5.6.css)  (../../../wp-content/themes/polite/assets/fonts/fonts.css@ver=1.4.0.css)  (../../../wp-content/themes/polite/css/font-awesome.min.css@ver=4.5.0.css)  (../../../wp-content/themes/polite/css/grid.min.css@ver=1.4.0.css)  (../../../wp-content/themes/polite/assets/css/slick.css@ver=1.8.1.css)  (../../../wp-content/themes/polite/assets/css/canvi.css@ver=1.4.0.css)  (../../../wp-content/themes/polite/style.css@ver=1.4.0.css)  (../../../wp-content/plugins/enlighter/cache/enlighterjs.min.css@ver=alHQQspbxGxjL5z.css)  (https://blog.christianperone.com/wp-json/)  (JSON) (https://blog.christianperone.com/wp-json/wp/v2/posts/5884)  (RSD) (https://blog.christianperone.com/xmlrpc.php?rsd)   (https://blog.christianperone.com/?p=5884)   (https://blog.christianperone.com/xmlrpc.php)  (index.html#content) Skip to content (https://blog.christianperone.com/) Terra Incognita  by Christian S. Perone         Search for: (Search …) ()   (Search)     Menu  (http://blog.christianperone.com) Home  (https://twitter.com/tarantulae) X  (https://bsky.app/profile/cperone.bsky.social) BlueSky (BSky)  (https://github.com/perone) Github  (https://www.linkedin.com/in/cperone) LinkedIn          (https://blog.christianperone.com/) Home    » (https://blog.christianperone.com/2026/) 2026    » (https://blog.christianperone.com/2026/09/) September    » (index.html) The systems that no one will test    »     (https://blog.christianperone.com/category/machine-learning/) Machine Learning   (https://blog.christianperone.com/tag/cybersec/) cybersec , (https://blog.christianperone.com/tag/machine-learning-2/) machine learning   The systems that no one will test (index.html) 28/09/2026   by (https://blog.christianperone.com/author/admin/) Christian S. Perone     This happened to me in 2020, and it has been on my mind again lately. During the worst of the pandemic, I found a vulnerability in a system that gave me access to the Brazilian federal system, and with that access I was able to retrieve information on any Brazilian (think of 200+ million people data). These records had essentially each person’s entire set of documents (ID, CPF, passport, place of birth, parents’ names, driver’s licence, home addresses, mobile and landline numbers, whether the person was in a witness protection programme, etc.), so you can get a sense of how serious this was and how much someone could do with this data. Honestly, I didn’t quite know how to react and I immediately phoned the agency responsible for the system (it wasn’t easy to find the right contact, you obviously don’t want to disclose that to the wrong person). When the person picked up my call, I identified myself and told him what I had found and what access I had. He was in disbelief at first, thinking it didn’t make much sense (and I understand his confusion, imagine someone calling you out of the blue and bringing a weird topic), and asked whether I was a civilian. I said yes, and a few hours later they got back in touch and asked me to explain the issue. By then I think they were a bit worried but still not believing much on it, and once I explained the problem, they probably realised very quickly that this was a major breach. They acknowledged the issue and fixed it very fast. I honestly don’t blame them: these same people were working really hard during the pandemic, and software is software. I was glad to have helped in some small way (at least I like to think that way). I never exfiltrated any data from this system, and I avoided talking about it later because it makes you look like a “hacker”, which is something stupid to say. So why am I talking about this ? Between 2020 and 2026, a lot changed, and ML began advancing at a crazy pace (and it still is), at perhaps the worst possible geopolitical moment. Fast-forward to 2026, and we’re seeing reports of cybersecurity incidents involving many different models, including the one OpenAI described in its (https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) technical report . Although I’m not sure the narrative that the model “escaped its safeguards ” makes much sense when OpenAI deliberately disabled classifiers and reduced safeguards (something many people weren’t aware of), these models are now clearly a big problem for nation-states. Even though safeguards are the labs’ choice, we need to acknowledge that the capability itself is real, and clearly not everyone will run evals (or, even worse, training) with proper safeguards enabled. I think training is even more complicated, because there it is also easier for the agent to figure out how to bypass/exploit safeguards. Following recent advances in mid-training, RLVR and long-horizon tasks, it is clear that over the past year (or more) many labs have been aggressively scaling RL environments, usind third-party companies, using models to build them, to synthesise them and provide their rewards, in a “self-improvement” loop (in quotes because the term is quite overloaded nowadays) that accelerates as fast as your rollout potential. That’s why I keep returning to the episode I described earlier: today it would be straightforward for me to build an RL environment that not only replicates the setup in which I found the vulnerability (which is not that different from the one in OpenAI’s report, where an agent leveraged SSRF through OpenAI’s internal Artifactory) but also to develop a curriculum for it. So I wonder: how many environments like this are frontier labs now building with support from cybersecurity companies ? What will happen to the many government systems that will never get the chance to be AI-pentested  ? The flaw I found didn’t require any extraordinary expertise, just attention to detail, and that is exactly what worries me. What will happen from now on, with agents that are far more capable, faster and easily scalable ? Just like the climate problem, ML is now a planetary problem, and as Yuk Hui mentioned in his book Machine and Sovereignty: For a Planetary Thinking  (which I recommend to anyone interested), we are still not thinking planetarily, but we must learn to, even if that takes considerable time . I think the only way forward is to first get past the polarisation between technophobia and technophilia (I wrote a bit about it (https://blog.christianperone.com/2025/01/notes-on-gilbert-simondons-on-the-mode-of-existence-of-technical-objects-and-artificial-intelligence/) here ) and start working on solutions. Obviously many people in frontier labs are already doing this work. But it seems to me an impossible task to find a solution while our thinking is guided by geopolitical interests. Unfortunately, emerging economies and developing countries will be the most vulnerable, as if it weren’t already enough that they seldom have access to compute. That is a very sad thing to think about. I also can’t help imagining that, somewhere, there is a very similar environment that reproduces what I found in 2020: almost the same system, almost the same flaw. Multiple agents enter it thousands of times a second, generating trajectories until one of them finds the flaw. The difference is that when this is put into use, in none of the attempts does anyone pick up the phone.  (https://www.facebook.com/sharer/sharer.php?u=https://blog.christianperone.com/2026/09/the-systems-that-no-one-will-test/)   (http://twitter.com/share?text=The%20systems%20that%20no%20one%20will%20test&url=https://blog.christianperone.com/2026/09/the-systems-that-no-one-will-test/)   (http://pinterest.com/pin/create/button/?url=https://blog.christianperone.com/2026/09/the-systems-that-no-one-will-test/&media=&description=The%20systems%20that%20no%20one%20will%20test)   (http://www.linkedin.com/shareArticle?mini=true&title=The%20systems%20that%20no%20one%20will%20test&url=https://blog.christianperone.com/2026/09/the-systems-that-no-one-will-test/)      Post navigation (https://blog.christianperone.com/2026/06/portable-ml-jobs-monitor-with-m5stack/) Portable ML jobs monitor with M5Stack       Leave a Reply (index.html#respond) Cancel reply   Your email address will not be published. Required fields are marked *   Comment *    Name ()   Email ()   Website ()   (Post Comment)  (5884)  (0)   (20f7b554d2)   Δ  (91)     This site uses Akismet to reduce spam. (https://akismet.com/privacy/) Learn how your comment data is processed.     Author   Christian S. Perone  Machine Learning Research Engineer London, UK      Search for: (Search …) ()   (Search)    Recent Posts (index.html) The systems that no one will test  (https://blog.christianperone.com/2026/06/portable-ml-jobs-monitor-with-m5stack/) Portable ML jobs monitor with M5Stack  (https://blog.christianperone.com/2026/06/where-the-wild-discovery-loops-are/) Where the wild Discovery Loops are  (https://blog.christianperone.com/2026/06/flow-matching-elites-with-nvidia-physicalai-autonomous-vehicles-dataset/) Flow Matching Elites with NVIDIA PhysicalAI Autonomous Vehicles dataset  (https://blog.christianperone.com/2026/01/auris-i-a-hackable-low-power-ai-pin/) Auris I – a hackable ultra low-power AI pin  (https://blog.christianperone.com/2025/11/slides-for-gemma3n-architecture-guide-talk/) Slides for Gemma3n architecture guide talk  (https://blog.christianperone.com/2025/07/diffusion-elites/) Diffusion Elites + World Models: surprisingly good, simple and embarrassingly parallel  (https://blog.christianperone.com/2025/05/torchstation-prototype-v1-gpus-panel/) TorchStation Prototype V1 – GPUs panel  (https://blog.christianperone.com/2025/04/vectorvfs-your-filesystem-as-a-vector-database/) VectorVFS: your filesystem as a vector database  (https://blog.christianperone.com/2025/01/notes-on-gilbert-simondons-on-the-mode-of-existence-of-technical-objects-and-artificial-intelligence/) Notes on Gilbert Simondon’s “On the Mode of Existence of Technical Objects” and Artificial Intelligence  (https://blog.christianperone.com/2024/11/the-geometry-of-data-part-ii/) The geometry of data: the missing metric tensor and the Stein score [Part II]  (https://blog.christianperone.com/2024/08/torch-titan-distributed-training/) Torch Titan distributed training code analysis  (https://blog.christianperone.com/2024/08/mmap-torch-np-jax-tf/) Memory-mapped CPU tensor between Torch, Numpy, Jax and TensorFlow    (http://creativecommons.org/licenses/by-nc/4.0/) (Creative Commons License)    This work is licensed under a (http://creativecommons.org/licenses/by-nc/4.0/) Creative Commons Attribution-NonCommercial 4.0 International License .           Tags (https://blog.christianperone.com/tag/announce/) announce (https://blog.christianperone.com/tag/arduino/) arduino (https://blog.christianperone.com/tag/article/) Article (https://blog.christianperone.com/tag/bayesian/) bayesian (https://blog.christianperone.com/tag/benford-law/) benford law (https://blog.christianperone.com/tag/c/) c (https://blog.christianperone.com/tag/cnn/) cnn (https://blog.christianperone.com/tag/convolutional-neural-networks/) convolutional neural networks (https://blog.christianperone.com/tag/covid/) covid (https://blog.christianperone.com/tag/deep-learning/) deep learning (https://blog.christianperone.com/tag/evolution/) evolution (https://blog.christianperone.com/tag/evolutionary-algorithms/) evolutionary algorithms (https://blog.christianperone.com/tag/feature-extraction/) feature extraction (https://blog.christianperone.com/tag/ga/) ga (https://blog.christianperone.com/tag/genetic-algorithm/) genetic algorithm (https://blog.christianperone.com/tag/genetic-algorithms/) Genetic Algorithms (https://blog.christianperone.com/tag/genetic-programming/) genetic programming (https://blog.christianperone.com/tag/hardware/) hardware (https://blog.christianperone.com/tag/image-processing/) Image Processing (https://blog.christianperone.com/tag/jit/) jit (https://blog.christianperone.com/tag/jython/) jython (https://blog.christianperone.com/tag/karl-popper/) karl popper (https://blog.christianperone.com/tag/language-model/) language model (https://blog.christianperone.com/tag/llm/) llm (https://blog.christianperone.com/tag/llvm/) LLVM (https://blog.christianperone.com/tag/machine-learning-2/) machine learning (https://blog.christianperone.com/tag/math/) Math (https://blog.christianperone.com/tag/matplotlib/) matplotlib (https://blog.christianperone.com/tag/modis/) modis (https://blog.christianperone.com/tag/news/) News (https://blog.christianperone.com/tag/nlp/) nlp (https://blog.christianperone.com/tag/philosophy/) Philosophy (https://blog.christianperone.com/tag/programming/) programming (https://blog.christianperone.com/tag/pyevolve/) Pyevolve (https://blog.christianperone.com/tag/python/) Python (https://blog.christianperone.com/tag/pytorch/) pytorch (https://blog.christianperone.com/tag/raspberry-pi-2/) raspberry pi (https://blog.christianperone.com/tag/science/) Science (https://blog.christianperone.com/tag/scikit-learn/) scikit.learn (https://blog.christianperone.com/tag/sigevolution/) sigevolution (https://blog.christianperone.com/tag/tf-idf/) tf-idf (https://blog.christianperone.com/tag/twitter/) twitter (https://blog.christianperone.com/tag/ubigraph/) ubigraph (https://blog.christianperone.com/tag/uncertainty/) uncertainty (https://blog.christianperone.com/tag/visualization/) visualization     (http://creativecommons.org/licenses/by-nc/4.0/) (Creative Commons License)    This work is licensed under a (http://creativecommons.org/licenses/by-nc/4.0/) Creative Commons Attribution-NonCommercial 4.0 International License .       Copyright All Rights Reserved 2025  (https://wordpress.org/) Proudly powered by WordPress  |  Theme: Polite by (https://www.templatesell.com/) Template Sell .      (index.html#) (Go to Top)      This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settings ACCEPT   Privacy & Cookies Policy     Close  Privacy Overview This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.     Necessary   Necessary  Always Enabled  Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.     Non-necessary   Non-necessary    Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.         SAVE & ACCEPT