(../../../www.blogger.com/static/v1/widgets/4165133002-widget_css_bundle.css) (../../favicon.ico) (a-jev-like-wrapper-for-llms-including.html) (Allan's Blog - Atom) (http://allanrbo.blogspot.com/feeds/posts/default) (Allan's Blog - RSS) (http://allanrbo.blogspot.com/feeds/posts/default?alt=rss) (Allan's Blog - Atom) (https://www.blogger.com/feeds/24624162/posts/default) (Allan's Blog - Atom) (http://allanrbo.blogspot.com/feeds/5391735760720655520/comments/default) (../../../blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgslbgarcdEUUkIJWtO6Hzy1tHhrkARR5gWxsFPfsPk02zgLqPXsCoxkEIHa4ymkDB3aF-21KS9F0f4ScH9KVz-iN9GJ6yyTDC4rh8AaPG2QiCbZXDUNkRnRYLl8YDcuC8FsX9mkm-A9sb5s9V86UyC3A_I1Ai19idZYTgW1g8ZBLvwYirK97hZSw/s1600/webcam.jpeg) Allan's Blog: A Jev-like wrapper for LLMs, including vision models (../../../www.blogger.com/dyncss/24624162/authorization.css@zx=dba96a70-d03e-4c66-9883-92cf29bfbe8f.css) (../../../www.blogger.com/dyncss/24624162/authorization.css@zx=dba96a70-d03e-4c66-9883-92cf29bfbe8f.css) (a-jev-like-wrapper-for-llms-including.html#main) skip to main | (a-jev-like-wrapper-for-llms-including.html#sidebar) skip to sidebar (http://allanrbo.blogspot.com/) Allan's Blog Allan Riordan Boll's blog Friday, September 25, 2026 A Jev-like wrapper for LLMs, including vision models (https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgslbgarcdEUUkIJWtO6Hzy1tHhrkARR5gWxsFPfsPk02zgLqPXsCoxkEIHa4ymkDB3aF-21KS9F0f4ScH9KVz-iN9GJ6yyTDC4rh8AaPG2QiCbZXDUNkRnRYLl8YDcuC8FsX9mkm-A9sb5s9V86UyC3A_I1Ai19idZYTgW1g8ZBLvwYirK97hZSw/s1369/webcam.jpeg) () I was intrigued by (https://docs.typesafe.ai/introduction) Jev and the self-hostable projects appearing around it, such as (https://huggingface.co/openjev/openjev) OpenJev and (https://github.com/TheoLeeCJ/SemIf-OpenJev) SemIf . Reading about them introduced me to a neat trick: reading an LLM's token probabilities. Apparently this is an old trick for some people. See e.g. (https://developers.openai.com/cookbook/examples/using_logprobs) OpenAI's logprobs cookbook . But it was new to me. I believe the basic idea is to write a prompt like this: State: My order arrived broken and I want a refund. Question: Which team should handle this? [A] billing [B] shipping [C] returns Answer with the letter of the best option only. Then add a few JSON request parameters to a compatible Chat Completions request: { "max_completion_tokens": 1, "logprobs": true, "top_logprobs": 20 } The LLM API will return the letter plus the model's log probabilities for alternative tokens. Repeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it. The fun part: this works with vision models too. Jev's (https://docs.typesafe.ai/introduction/quickstart) documented request format currently describes only text/JSON state. I added an attachments field for images for my local experiments. My example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? With Gemma 4 12B on my RTX 3090, I get around 1 frames per second , with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn't make any effort to avoid the cost of a separate connection through their system per question per frame. Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text. Here's the standalone Python example (OpenCV is just used for convenient access to the webcam, not for any actual computer vision): #!/usr/bin/env -S uv run --script # /// script # dependencies = ["opencv-python"] # /// """Preview and score webcam frames with llama.cpp or OpenAI. uv run webcam.py uv run webcam.py https://api.openai.com/v1 gpt-6-luna OpenAI reads OPENAI_API_KEY. """ import argparse import base64 import concurrent.futures import datetime import json import math import mimetypes import os import pathlib import time import urllib.parse import urllib.request import cv2 # attachments is our custom addition to the Jev request format. data = json.loads(""" { "state": "Inspect this webcam frame. Judge only what is visibly present.", "attachments": [], "questions": { "person": { "type": "noul", "instructions": "Is a person visible?" }, "plant": { "type": "noul", "instructions": "Is a plant visible?" }, "setting": { "type": "choice", "instructions": "Where is the camera?", "criteria": { "indoors": null, "outdoors": null, "unclear": null } }, "light": { "type": "score", "instructions": "How bright is the scene?", "criteria": [ "dark", "dim", "bright" ] } } } """) def score(data, url, model): state = data["state"] if not isinstance(state, str): state = json.dumps(state) # Attachments are our extension to the Jev-style request format: # image file paths or base64 data URLs. Load them once for all questions. images = [] for attachment in data.get("attachments", []): if attachment.startswith("data:image/"): images.append(attachment) continue path = pathlib.Path(attachment).expanduser() mime_type, _ = mimetypes.guess_type(path) if mime_type not in {"image/png", "image/jpeg", "image/webp", "image/gif"}: raise ValueError(f"Unsupported image file: {path}") encoded = base64.b64encode(path.read_bytes()).decode() images.append(f"data:{mime_type};base64,{encoded}") # Send the API key only to OpenAI. is_openai = urllib.parse.urlsplit(url).hostname == "api.openai.com" headers = {"Content-Type": "application/json"} if is_openai: headers["Authorization"] = "Bearer " + os.environ["OPENAI_API_KEY"] answers = {} for name, question in data["questions"].items(): # Represent choices, booleans, and ordinal levels as lettered options. if question["type"] == "choice": options = question["criteria"] elif question["type"] == "noul": options = {"true": None, "false": None} | question.get("criteria", {}) elif question["type"] == "score": options = {str(i): description for i, description in enumerate(question["criteria"])} else: raise ValueError(f"Unknown question type: {question['type']}") if not 2 <= len(options) <= 20: raise ValueError("Provide 2 to 20 criteria per question.") letters = "ABCDEFGHIJKLMNOPQRST"[:len(options)] # Ask for a single option letter, so its logprob represents that option. instructions = question["instructions"] if not isinstance(instructions, str): instructions = json.dumps(instructions) lines = [f"State:\n{state}\n\nQuestion: {instructions}\nOptions:"] for letter, (key, description) in zip(letters, options.items()): line = f"[{letter}] {key}" if description is not None: line += f": {description}" lines.append(line) prompt = "\n".join(lines) + "\n\nAnswer with the letter of the best option only." # OpenAI needs Responses for enough alternatives; llama.cpp needs Chat for logprobs. # top_p=1 avoids pruning alternatives. if is_openai: endpoint = "/responses" content = [{"type": "input_text", "text": prompt}] content.extend({"type": "input_image", "image_url": image} for image in images) body = { "model": model, "input": [{"role": "user", "content": content}], "reasoning": {"effort": "none"}, "max_output_tokens": 16, "top_p": 1, "top_logprobs": 20, "include": ["message.output_text.logprobs"], } else: endpoint = "/chat/completions" content = [{"type": "text", "text": prompt}] content.extend({"type": "image_url", "image_url": {"url": image}} for image in images) body = { "model": model, "messages": [{"role": "user", "content": content}], "max_completion_tokens": 1, "temperature": 0, "reasoning_effort": "none", "logprobs": True, "top_logprobs": 1024, } # Send the request and read the first output token's alternatives. request = urllib.request.Request( url.rstrip("/") + endpoint, headers=headers, data=json.dumps(body).encode(), ) with urllib.request.urlopen(request) as response: result = json.load(response) if is_openai: message = next(item for item in result["output"] if item["type"] == "message") candidates = message["content"][0]["logprobs"][0]["top_logprobs"] else: candidates = result["choices"][0]["logprobs"]["content"][0]["top_logprobs"] logprobs = {item["token"]: item["logprob"] for item in candidates} # Normalize the returned option scores; missing options initially get zero. missing = [letter for letter in letters if letter not in logprobs or logprobs[letter] <= -9999] if len(missing) == len(letters): raise ValueError("API did not return usable scores for any option") peak = max(logprobs[letter] for letter in letters if letter not in missing) weights = [math.exp(logprobs[letter] - peak) if letter not in missing else 0 for letter in letters] total = sum(weights) # An omitted token cannot outrank the last returned alternative. # Allow zero only when their combined normalized probability is below 1e-6. if missing: cutoff = min(value for value in logprobs.values() if value > -9999) missing_weight = len(missing) * math.exp(cutoff - peak) if missing_weight / (total + missing_weight) >= 1e-6: raise ValueError(f"API omitted non-negligible option scores for: {', '.join(missing)}") probabilities = {key: weight / total for key, weight in zip(options, weights)} # Return the winning choice, probability of true, or expected ordinal level. if question["type"] == "choice": answers[name] = { "type": "choice", "choice": max(probabilities, key=probabilities.get), "probabilities": probabilities, } elif question["type"] == "noul": answers[name] = {"type": "noul", "noul": probabilities["true"]} else: answers[name] = { "type": "score", "score": sum(int(key) * probability for key, probability in probabilities.items()), "legend": options, "probabilities": probabilities, } return {"answers": answers} # Choose the server and model before opening the camera. parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("url", nargs="?", default="http://localhost:8060/v1") parser.add_argument("model", nargs="?", default="gemma-4-12b") args = parser.parse_args() # Point OpenCV's bundled Qt at the installed system fonts. os.environ["QT_QPA_FONTDIR"] = "/usr/share/fonts/truetype/noto" # Open the default Linux webcam with a small capture buffer. camera = cv2.VideoCapture(0, cv2.CAP_V4L2) if not camera.isOpened(): raise RuntimeError("Could not open /dev/video0") camera.set(cv2.CAP_PROP_BUFFERSIZE, 1) print(f"Webcam -> {args.model}. Noul: yes %; score: value/max. Ctrl-C or Esc to stop.", flush=True) print(f"{'time':<8}" + "".join(f"{name:>10}" for name in data["questions"]) + f"{'fps':>10}", flush=True) # Preview continuously while a background worker scores one frame at a time. executor = concurrent.futures.ThreadPoolExecutor(max_workers=1) pending = None try: while True: ok, frame = camera.read() if not ok: raise RuntimeError("Could not read a webcam frame") cv2.imshow("Webcam", frame) if cv2.waitKey(1) == 27 or cv2.getWindowProperty("Webcam", cv2.WND_PROP_VISIBLE) < 1: break # Print a completed result, then submit the latest frame. if pending is not None: if not pending.done(): continue result = pending.result() columns = [] for name in data["questions"]: answer = result["answers"][name] if answer["type"] == "noul": value = f"{answer['noul']:.1%}" elif answer["type"] == "choice": value = answer["choice"] else: value = f"{answer['score']:.2f}/{len(data['questions'][name]['criteria']) - 1}" columns.append(f"{value:>10}") columns.append(f"{1 / (time.perf_counter() - started):>10.2f}") print(captured + "".join(columns), flush=True) # Measure throughput for evaluated frames, including image encoding. started = time.perf_counter() captured = datetime.datetime.now().strftime("%H:%M:%S") ok, jpeg = cv2.imencode(".jpg", frame) if not ok: raise RuntimeError("Could not encode the webcam frame") image = "data:image/jpeg;base64," + base64.b64encode(jpeg.tobytes()).decode() data["attachments"] = [image] pending = executor.submit(score, data, args.url, args.model) except KeyboardInterrupt: print("\nStopped.") finally: camera.release() cv2.destroyAllWindows() executor.shutdown() The script handles the API differences: llama.cpp uses Chat Completions and OpenAI uses Responses to get it to show alternatives. I ran Gemma 4 12B QAT through llama.cpp. On Linux with NVIDIA drivers, curl , zstd , and uv installed: # Model (~7 GB) and multimodal projector (~175 MB). mkdir -p ~/models/gemma-4-12b/ cd ~/models/gemma-4-12b/ curl -fL -C - -o gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/gemma-4-12b-it-qat-q4_0.gguf curl -fL -C - -o mmproj-gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf # Standalone llama.cpp binary for RTX 3090 (CUDA architecture 86). curl -fL -o llama.zst https://huggingface.co/buckets/ggml-org/install.sh/resolve/b11160/x86_64/linux/cuda/86/llama-app.zst mkdir -p ~/bin/ zstd -d llama.zst -o ~/bin/llama chmod +x ~/bin/llama ~/bin/llama serve --models-dir ~/models/ --port 8060 Save the Python example as webcam.py . In another terminal, from that directory: uv run webcam.py http://localhost:8060/v1 gemma-4-12b # Or use OpenAI, with OPENAI_API_KEY set in your environment. uv run webcam.py https://api.openai.com/v1 gpt-6-luna Posted by (https://www.blogger.com/profile/11850694461776134997) (author profile) Allan at (a-jev-like-wrapper-for-llms-including.html) (permanent link) (2026-09-25T21:19:00-07:00) 21:19 (https://www.blogger.com/post-edit.g?blogID=24624162&postID=5391735760720655520&from=pencil) (Edit Post) () No comments: Post a Comment (https://www.blogger.com/comment/frame/24624162?po=5391735760720655520&hl=en&saa=85391&origin=http://allanrbo.blogspot.com) (http://allanrbo.blogspot.com/2023/03/peer-to-peer-voice-over-ip-call-with.html) (Older Post) Older Post (http://allanrbo.blogspot.com/) Home Subscribe to: (http://allanrbo.blogspot.com/feeds/5391735760720655520/comments/default) Post Comments (Atom) Blog Archive ▼ (http://allanrbo.blogspot.com/2026/) 2026 (1) ▼ (http://allanrbo.blogspot.com/2026/09/) September (1) (a-jev-like-wrapper-for-llms-including.html) A Jev-like wrapper for LLMs, including vision models ► (http://allanrbo.blogspot.com/2023/) 2023 (1) ► (http://allanrbo.blogspot.com/2023/03/) March (1) ► (http://allanrbo.blogspot.com/2021/) 2021 (5) ► (http://allanrbo.blogspot.com/2021/08/) August (1) ► (http://allanrbo.blogspot.com/2021/04/) April (2) ► (http://allanrbo.blogspot.com/2021/02/) February (2) ► (http://allanrbo.blogspot.com/2020/) 2020 (6) ► (http://allanrbo.blogspot.com/2020/07/) July (1) ► (http://allanrbo.blogspot.com/2020/06/) June (4) ► (http://allanrbo.blogspot.com/2020/01/) January (1) ► (http://allanrbo.blogspot.com/2018/) 2018 (2) ► (http://allanrbo.blogspot.com/2018/06/) June (1) ► (http://allanrbo.blogspot.com/2018/05/) May (1) ► (http://allanrbo.blogspot.com/2017/) 2017 (2) ► (http://allanrbo.blogspot.com/2017/11/) November (1) ► (http://allanrbo.blogspot.com/2017/05/) May (1) ► (http://allanrbo.blogspot.com/2016/) 2016 (4) ► (http://allanrbo.blogspot.com/2016/09/) September (2) ► (http://allanrbo.blogspot.com/2016/05/) May (1) ► (http://allanrbo.blogspot.com/2016/03/) March (1) ► (http://allanrbo.blogspot.com/2015/) 2015 (2) ► (http://allanrbo.blogspot.com/2015/06/) June (1) ► (http://allanrbo.blogspot.com/2015/03/) March (1) ► (http://allanrbo.blogspot.com/2014/) 2014 (4) ► (http://allanrbo.blogspot.com/2014/08/) August (3) ► (http://allanrbo.blogspot.com/2014/06/) June (1) ► (http://allanrbo.blogspot.com/2013/) 2013 (2) ► (http://allanrbo.blogspot.com/2013/06/) June (1) ► (http://allanrbo.blogspot.com/2013/01/) January (1) ► (http://allanrbo.blogspot.com/2012/) 2012 (7) ► (http://allanrbo.blogspot.com/2012/07/) July (4) ► (http://allanrbo.blogspot.com/2012/06/) June (1) ► (http://allanrbo.blogspot.com/2012/04/) April (1) ► (http://allanrbo.blogspot.com/2012/02/) February (1) ► (http://allanrbo.blogspot.com/2011/) 2011 (2) ► (http://allanrbo.blogspot.com/2011/12/) December (2) ► (http://allanrbo.blogspot.com/2010/) 2010 (2) ► (http://allanrbo.blogspot.com/2010/02/) February (1) ► (http://allanrbo.blogspot.com/2010/01/) January (1) ► (http://allanrbo.blogspot.com/2009/) 2009 (2) ► (http://allanrbo.blogspot.com/2009/12/) December (1) ► (http://allanrbo.blogspot.com/2009/10/) October (1) ► (http://allanrbo.blogspot.com/2008/) 2008 (2) ► (http://allanrbo.blogspot.com/2008/12/) December (2) (../../../acoby.com/~allan/syntaxhighlighter/styles/SyntaxHighlighter.css)