What do you do when you view a diff, and what you see there doesn't match your editor?
Edit: I know emacs can probably view diffs. That's not the point, unless you expect every window to be the same width, and every tool to wrap everything exactly the same.
We should fix the diff tools too while we are at it. Why are we diffing by lines and not doing a semantic diff of the code? If the compiler doesn't care about it, why should the diff?
Raw text is a limited medium, and working at the level of plain text analysis rather than semantic analysis does limit how much our tools can achieve. The UNIX philosophy of having many small, text-based tools and chaining them together represents a common platform, but it’s also a least common denominator platform.
As long as we allow that glass ceiling to remain, the best our tools will ever do is push it incrementally higher, one tiny step at a time. If we want to make big leaps, we’re going to have to sacrifice some of that generality so we can use more powerful but specific tools. Unfortunately, that means that any new programming language wanting to take on the established standards needs not only a compiler/interpreter, but also the rest of the ecosystem: a comprehensive tool chain, ample library coverage, documentation and training resources, and so on. It is also, almost inevitably, going to need some standardised way of bridging the gap to today’s established languages for interoperability and backward compatibility purposes.
This is why I think the view that programming languages should be designed optimally for humans is short-sighted, and I suspect most of the big success stories in the coming years will be languages that were designed with clean semantics and easy parsing in mind. Those languages will better support building that surrounding ecosystem, and a good language with a good ecosystem is more practically useful than a slightly better language with a limited ecosystem.
> working at the level of plain text analysis rather than semantic analysis does limit how much our tools can achieve.
We can add semantic analysis. I have no problem with that. But removing or harming the ability to do text analysis might not stop us from using more sophisticated tools, but it does stop us from putting tools together quickly in a way not previously predicted. This is why Unix is so powerful. If you haven't read The Art of Unix Programming, I urge you to do so. It puts the argument and examples forward far more convincingly than I can.
Would moving to semantic tools harm the ability to use our existing text-based tools? I'd say so. A simple tool like diff works better, for example, if you take your big list of Python imports and put each one on a separate line, and keep them sorted. Patching works smoother this way too, reducing the likelyhood of the need for manual conflict resolution. If we eliminate doing this kind of arrangement by hand, and instead start relying on a semantic editor, we'll lose this ability. I have yet to see a tool that does semantic diffs, patches and merges better than diff, patch and git do.
But removing or harming the ability to do text analysis [...] does stop us from putting tools together quickly in a way not previously predicted.
Well, OK, but we’ve been using these text-based tools for a year or two now, and I don’t see many radical advances taking place in how we use or combine them. Are you sure you’re not chasing an illusion here?
The freeform text-based tools represent a great deal of flexibility, to be sure, but they were conceived at a time when flexible text manipulation was about as much as one could hope for. Today, we can do more.
I have yet to see a tool that does semantic diffs, patches and merges better than diff, patch and git do.
As long as everything is limited to manipulation of freeform text files, perhaps you never will. That doesn’t mean better tools aren’t possible; it just means they aren’t possible within the constraints you’re choosing to impose.
> Well, OK, but we’ve been using these text-based tools for a year or two now, and I don’t see many radical advances taking place in how we use or combine them. Are you sure you’re not chasing an illusion here?
I'm not claiming recent advances. I'm claiming existing power that has been around for decades, which we would lose if we compromised the text tooling available today.
Have you read TAOUP? Do you understand the extent of the power that existing text tooling gives us today? Are you experienced in the advanced use of the existing tools, so you are able to make comparisons about their power?
> As long as everything is limited to manipulation of freeform text files, perhaps you never will. That doesn’t mean better tools aren’t possible; it just means they aren’t possible within the constraints you’re choosing to impose.
This is backwards. We move forward when people show how it can be done. Please show us how we can improve diffs, patches and merges by moving to semantic data structures over text, without compromising any existing capabilities. Even just illustrating specifics of how these tools might work, rather than implementing them, will do something for your argument. The onus is on you.
I'm claiming existing power that has been around for decades, which we would lose if we compromised the text tooling available today.
Why would we lose it? The power of those tools isn’t in a particular executable, it’s in the algorithms they embody. For example, it is useful to compare two text streams reasonably efficiently and identify differences. How those differences are then presented obviously matters, but if you’ve got the algorithms and the ideas underlying them, producing a new tool to apply those ideas in a different context is the easy part.
The only significant difference I see is that if you made a major change, for example adopting a more structured storage model or using some sort of action/history analysis to better capture a programmer’s intent, then you would have more data to use in your algorithms, and you might therefore be able to present more interesting results.
Have you read TAOUP? Do you understand the extent of the power that existing text tooling gives us today? Are you experienced in the advanced use of the existing tools, so you are able to make comparisons about their power?
Yes, though I find your emphasis on that one book a little surprising. For one thing, the UNIX philosophy was established for decades before Raymond wrote that particular work. For another, I seem to recall that he gives examples of both text and binary formats being useful in the book. I don’t think his point was that text formats were good and non-text ones bad; I think he was arguing that things like adaptability and composability were good and that flexible and standardised formats helped to achieve those things.
We move forward when people show how it can be done.
Right, so why aren’t the programming language community picking up on decades of research and industrial progress with databases and HCI? Programming languages and the related tools are, fundamentally, just a user interface to design and control a complex, highly structured set of data.
Please show us how we can improve diffs, patches and merges by moving to semantic data structures over text, without compromising any existing capabilities.
You’re begging the question, by starting from the position that having an equivalent to today’s text-based diffs, patches and merge tools is desirable. I don’t think that is necessarily true.
As a programmer, I want to be able to specify how my software should work, and I want to be able to explore and modify that specification effectively, and I need to be able to do these things in collaboration with others. My claim is that to do those things much better than we do them today, we may need to move to a different representation than freeform text and then build new tools that are designed to solve our problems in terms of that new representation.
The problem is that there is so much momentum behind text-based formats today that we are effectively stuck around a local maximum. No one individual could possibly meet your challenge today, and I’m sure you were well aware of that when you made it. That doesn’t mean the community as a whole couldn’t do it, but it would need some serious collaboration, which realistically means one of the heavyweight organisations with the resources to bootstrap a whole new software development ecosystem would need to throw its weight behind such a project. Unfortunately, most if not all such organisations are commercial in nature, and the commercial incentives don’t align with moving in that direction.
The same diff tool works for text files, latex and other markup files, for any computer language, and so on. I can write downstream tools that take the output of the diff and do further processing, without worrying about what the output might look like for brainfuck. And how long would we have to wait for Torvalds to add a C++ or Java extension to git? :)
Good question. When I do a diff from within emacs, using ediff, it "just works" -- diffs are shown side-by-side w/ the same wrapping applied. (Ediff is also good at showing you what changed within the line.)
If I'm diffing in an external tool (like gitx, or just on the command line), then I just see (possibly) long lines. So far, it hasn't been an issue for me. Obviously, diff, as a line-oriented protocol, breaks down as lines get really long -- but it's still just code, so it's not like my lines are ever insanely-long. :)
I also use a mode that highlights my current (physical) line in the file, so I still tend to have a very good sense of what the physical line in the file is, despite the wrapped visual display. E.g., http://imgur.com/a/wAXHJ
One of the oft-touted benefits of wrapping lines at (say) 80 chars is it makes it easy to do side-by-side viewing of files -- using dynamic wrapping gives you this same benefit, but even more so, since you can heads-up different files at whatever width your current display happens to have. (Or however many files you want to have side-by-side.)
Also, there's a nice side-benefit to diffs, which is you don't get the noise that comes from a change that forces a manual rewrapping -- e.g., maybe I decide my variable "id" was too generic, so I change it to "frobnackId", but then this pushes some function call over the 80-character limit. If I'm manually rewrapping, I get a weird diff of multiple lines changing over the rewrap.
(On the other hand, a definite down-side is if you do find yourself using an editor that can't wrap nicely, code with long lines can be quite annoying.)
Vim shows diffs with wrapped lines just fine as far as I can tell. I have it set up to show both versions side by side, with changed lines highlighted and the particular change on each line also highlighted. I can wrap one or both, depending on how wide I make either.
Can you explain what problem you think might occur? Its not clear to me what problems you anticipate.
Humans are set up to see differences visually when they see patterns. If the way that lines wrap depends on the width of your window, then it becomes hard to match up the output visually from windows of different widths.
An example: I type "git diff" into one window, and look for some particular change area in my editor in a different window. If the wrap and alignments depend on the widths of my windows, then they won't match unless the widths of my windows also match. And if I have to make them match, then the original ideal of "make your window however wide you want and it'll just look right" is defeated.
Can't say I experience this problem, the highlighting of relevant lines takes care of this. Human eyes are terrible at tracking across long lines of text without jumping up or down a line (this is why splitting text into columns is common in large books). On the other hand jumping between regions of color is easy.
Furthermore, I don't think this is true: "And if I have to make them match, then the original ideal of "make your window however wide you want and it'll just look right" is defeated." Being able to resize both, with the constraint that they must both be kept in sync (which to be clear, I don't consider necessary), is better than the alternative: no effective resizing at all.
You should probably look into integrating git-diff with your editor though (use git-difftool).
Edit: I know emacs can probably view diffs. That's not the point, unless you expect every window to be the same width, and every tool to wrap everything exactly the same.