In general when you try a new tool or methodology you tend to start with a small class to see the results first.

This is not an AI tool, this is a CLI that has very verbose output and documentation.

It can be used by human or by AI agents.

I experiment the same with other mechanisms, and CLI are as effective - if not more effective - than MCP.

Granted, having access to AI I would use AI to run it. But nothing is stopping a manual, human centric, use.

I believe more tools should be written like that.

You don't need AI for it.

You can just install the tool and use it. It is a CLI with very verbose output. Verbose output is good for both humans and AI.

I created something similar, but instead of final oral examination, we do homework.

The student is supposed to submit a whole conversation with an LLMs.

The LLM is prompted to answer a question or resolve a problem, and the LLM is there to assist. The LLM is instructed to never reveal the answer.

More interesting is the concept that the whole conversation is available to the instructor for grading. So if the LLMs makes mistake, or give away the solution, or if the student prompt engineer around it. It is all there and the instructor can take the necessary corrective measures.

87% of the students quite liked it, and we are looking forward to doubling the students that will be using it next quarter.

Overall, we are looking for more instructor to use it. So if you are interested in it please get in touch.

More info on: https://llteacher.blogspot.com/

Good that at least you aren't forcing the student to sign up for these very exploitative services.

I'm still somewhat concerned about exposing kids to this level of sycophancy, but I guess it will be done with or without using it in education directly.

The perspective from an educator is quite concerning indeed.

Students are very simply NOT doing the work that is require to learn.

Before LLMs, homeworks were a great way to force students to approach the material. Students did not have any other way to get an answer, so they were forced to study and come up with an answer to the homeworks. They could always copy from classmates, but that was considered quite negatively.

LLMs change this completely. Any kind of homework you could assign undergraduates classes are now completed in less than 1 second, for free, by LLMs.

We start to see PERFECT homeworks submitted by students who could not get a 50% grade in classes. Overall grades went down.

This is a common pattern with all the educators I have been talking with. Not a single one has a different experience.

And, I do understand students. They are busy, they may not feel engaged by all the classes, and LLMs are a way too fast solution for getting homeworks done and free up some time.

But it is not helping them.

Solutions like this are to force students to put the correct amount of work in their education.

And I would love if all of this would not be necessary. But it is.

I come from an engineering school in Europe - we simply did not have homework. We had frontal classes and one big final exams. Courses in which only 10% of the class would pass were not uncommon.

But today education, especially in the US, is different.

This is not forcing student to use LLMs. We are trying to force student to think and do the right thing for them.

And I know it sounds very paternalistic - but if you have better ideas, I am open.

I think it's a mix of a few things:

- The stuff being covered in high school is indeed pretty useless for most people. Not all, but most, and it is not that irrational for many to actually ignore it.

- The reduction in social mobility decreasing the motivation for people to work hard for anything in general, as they get disillusioned.

- The assessment mechanisms being easily gamed through cheating doesn't help.

It's probably time to re-evaluate what's taught in school, and what really matters. I'm not that anti-school but a lot of the homework I've experienced simply did not have to be done in the first place, and LLM is exposing that reality. Switching to in-person oral/written exams and only viewing written works as supplementary, I think, is a fair solution for the time being.

I will be crucified by this, but I think you are doing it wrong.

I would split it in 2 steps.

First, just move it to svelte, maintain the same functionality and ideally wrap it into some tests. As mentioned you want something that can be used as pass/no-pass filter. As in yes, the code did not change the functionality.

Then, apply another pass from Svelte bad quality to Svelte good quality. Here the trick is that "good quality" is quite different and subjective. I found the models not quite able to grasp what "good quality" means in a codebase.

For the second pass, ideally you would feed an example of good modules in your codebase to follow and a description of what you think it is important.

I am toying with something similar.

However my approach would be to use duckdb and S3 over lambda.

Leaving many of the concerns to the infrastructure. Like basically no OOM. No need to manage servers.

With my partner we have been working to invert the overall model.

She started grading conversation than the students have with LLMs.

From the question that the students ask, it is obvious who knows the material and who is struggling.

We do have a custom setup, so that she creates an homework. There is a custom prompt to avoid the LLM answering the homework question. But thats pretty much it.

The results seems promising, with students spending 30m or so going back and forth with the LLMs.

If any educator wants to Ty or is interested in more information, let me know and we can see how we collaborate.

This makes some sense, but my first question would be how do you define a clear, fair grading rubric? Second, this sounds like it could work for checking who is smart, but can it motivate students to put in work to learn the material?

Just to echo the point of MCP, they seem cool, but in my experience just using a CLI is orders of magnitude faster to write and to debug (I just run the CLI myself, put test in the code, etc...)

Jup and it doesn't bloat the context unnecessarily. The agent can call --help when it needs it. Just imagine a kubectl MCP with all the commands as individual tools, doesn't make any sense whatsoever.

> and it doesn't bloat the context unnecessarily.

And, this is why I usually use simple system prompts/direct chat for "heavy" problems/development that require reasoning. The context bloat is getting pretty nutty, and is definitely detrimental to performance.

Do you have any information e.g. blog posts on this pattern?

The point of this stuff is to increase reliability. Sure the LLM has a good chance of figuring out the skill by itself, the idea is that its less likely to fuck up with the skill though. This is an engineering advancement that makes it easier for businesses to rely on LLMs for routine stuff with less oversight.

This is absolutely neat.