External module resolution for non-Typescript packages

[poelstra]

Good progress is made on supporting the case of packages that carry their own typings in #2338.

That proposal doesn't currently deal with modules that do not carry their own typings, a.k.a. all modules that need typings from DefinitelyTyped.

This issue is to investigate possibilities of, and propose a (simple) solution for preventing problems when using (multiple versions of) external modules together.

TL;DR

1. No longer use declare module "..." { } in typings for external modules
2. Add an extra step to the resolution logic of External module resolution logic #2338 to look up this type of external typings

Example scenario

• myprogram
  • mylib
    • myutils@1.0
  • myotherlib
    • myutils@2.0

Suppose that both mylib and myotherlib export a function, that returns a value of a type defined in myutils.

Existing scenario

Likely, both mylib and myotherlib will each have a /// <reference path="./typings/tsd.d.ts" /> line, which again has a /// <reference path="./myutils/myutils.d.ts" /> line.

Problem

Both libs carry their own version of myutils.d.ts, which works fine as long as they are not used together.

However, when they are used together (as in this example), all the /// <reference ...> lines will (need to) be 'merged', which essentially means the declare module "myutils" { ... } from both mylib and myotherlib will end up in the same 'global module name namespace'.

This gives a compiler error, and even if the compiler would somehow allow it, would make it impossible to refer to either version of myutils.

Additionally, tools like dts-generator e.g. include the /// <reference> line inside of the declare blocks, which get ignored by the compiler. This leads to unknown-references when trying to use such a typing, and will typically be solved by adding e.g. a myutils/myutils.d.ts to myprogram/typings/tsd.d.ts, which will be wrong for one of the libs.

Solution

External modules have the nice property that there is no 'global namespace' (except the filesystem, maybe), and basically every reference is 'local'.

It makes sense to use the same referencing scheme that's used for resolving 'local' external modules (within a package) for 'normal' external modules (different npm packages).

Basically, the idea consists of:

• No longer using declare module "..." { } in typings for external modules (to prevent the global namespace clash)
• Adding an extra step to the resolution logic of External module resolution logic #2338 to look up this type of external typings (e.g. in typings/ folder)

Advantages

• Multiple versions of a package can co-exist in the same program/library
• /// <reference> lines are no longer needed in most cases (except probably for e.g. importing Mocha's it() and other 'true' globals)
• One format to rule them all: external module typings are already generated without declare module "..." { } by tsc, so all 'sorts' of external module typings will look the same
• Gradual upgrade path: the tsd.d.ts-way will still work for packages that do still use declare module "..." { } (although it will still have the name-clash problem etc.)

Disadvantages

• Typings on e.g. DefinitelyTyped will (gradually) need to be replaced with their non-wrapped version
• Extra lookup step in the compiler
• Location of these external typings (a.k.a. include path) needs to be configured somewhere and/or sensible default value chosen

Example 'new-style' typings

Most hand-crafted typings will typically look like:

/// myutils.d.ts
export interface BarType {
    bar: number;
}
export declare function something(): BarType;

But just to make things more interesting, suppose the typings of myutils look like:

/// dist/bar.d.ts
export interface BarType {
    bar: number;
}

/// dist/foo.d.ts
import bar = require("./bar");
export declare function something(): bar.BarType;

Nothing special here, this is what tsc generates today.
(Note: dist/ may be because someone used CoffeeScript, not TypeScript, right? :))

Now, the typings dir on e.g. DefinitelyTyped could look like:

DefinitelyTyped/myutils/latest/dist/bar.d.ts
DefinitelyTyped/myutils/latest/dist/foo.d.ts
DefinitelyTyped/myutils/1.0/dist/bar.d.ts
DefinitelyTyped/myutils/1.0/dist/foo.d.ts

(Or maybe 2.0 instead of latest, ideas welcome.)

And both mylib and myotherlib would have:

typings/myutils/dist/bar.d.ts
typings/myutils/dist/foo.d.ts

Looking up .d.ts

Given e.g. import { something } from "myutils" the compiler could use myutils/package.json's main property to get to ./dist/foo.js, which in turn would resolve to dist/foo.d.ts.

Nice and simple: no need to have support from the authors of myutils, no /// <reference> lines.
And it even supports the import bar = require("myutils/dist/bar") case.

Open questions

• Use one typings dir for both 'wrapped' and 'unwrapped' typings?
• First do a lookup for external modules already declare'ed, e.g. through tsd.d.ts? If so, would probably make first bullet usable, nice for gradual upgrade path of DT typings.
• Most DT typings will not want to mimic exact filesystem structure of package, using e.g. myutils/latest/index.d.ts will probably work out-of-the-box with external module resolution logic, but myutils/latest/myutils.d.ts or even myutils/myutils.d.ts may be easier for filename in editor tab?

Good idea? Bad idea?

[basarat]

My hacky suggestion:

• myprogram
  • mylib
    • myutils@1.0
  • myutils@2.0

In this case again if we only read myutils@2.d.ts and mylib.d.ts then there would be no issue. But since mylib.d.ts references myutils@1.d.ts we will have a conflict. My proposal is a conflict resolution algorithm such that:

• myprogram
  • mylib
    • myutils@1.0
  • myutils@2.0

We only read the following files into the compilation context of myprogram : [mylib.d.ts,myutils@2.d.ts]

---

Note in:

• myprogram
  • mylib
    • myutils@1.0
  • myotherlib
    • myutils@2.0

Compilation context simply picks the latest versions, so : [mylib.d.ts,myutils@2.d.ts]

Hacky

Note that if you are using the browser (single version of each js) then this is probably what you want.
If you are using Node then even though example 2 will work fine we are forcing myutils@2 down the throat of mylib.

[poelstra]

@basarat Thanks for looking into this! 🌹

In my mail I saw a different post than the one above, seems you've editted it :)

Anyway, you made a valid point, so let me comment on that:

This gives a compiler error, and even if the compiler would somehow allow it,
would make it impossible to refer to either version of myutils.

myutils will (and should) not be usable in myprogram. So they can both have their declare module
"myutils" and if we don't read that in myprogram then myprogram should compile just fine.

I may have phrased it a little unclear. I agree that myutils should not be usable in myprogram directly.
What I meant was that the require("myutils") (or import ... from "myutils";) statements that will be present in mylib and myotherlib can only be resolved to a single declare module "myutils" {}.

Re your 'hacky solution': I don't think a conflict resolution like you're proposing is going to work, and may lead to uncompilable code.

Example of it going wrong using 'wrapped' external modules:

/// mylib/typings/myutils.d.ts (i.e. the 1.0 typings)
declare module "myutils" {
  export interface SomeOldInterface { ... }
  export declare function something(): SomeOldInterface;
}

/// mylib/dist/mylib.d.ts
declare module "mylib" {
  import myutils = require("myutils");
  export declare function myfunc(): myutils.SomeOldInterface;
}

/// myotherlib/typings/myutils.d.ts (i.e. the 2.0 typings)
declare module "myutils" {
  export interface SomeNewerInterface { ... }
  export declare function something(): SomeNewerInterface;
}

/// myotherlib/dist/myotherlib.d.ts
declare module "myotherlib" {
  import myutils = require("myutils");
  export declare function myotherfunc(): myutils.SomeNewInterface;
}

I didn't actually try to compile this, but I'm sure we'll get a compiler error on the return value of either myfunc() or myotherfunc() not existing (depending on the order of your resolution).

Contrast this with the following case:

/// mylib/typings/myutils.d.ts
export interface SomeOldInterface { ... }
export declare function something(): SomeOldInterface;

/// mylib/dist/mylib.d.ts
import myutils = require("myutils");
export declare function myfunc(): myutils.SomeOldInterface;

/// myotherlib/typings/myutils.d.ts
export interface SomeNewerInterface { ... }
export declare function something(): SomeNewerInterface;

/// myotherlib/dist/myotherlib.d.ts
import myutils = require("myutils");
export declare function myotherfunc(): myutils.SomeNewInterface;

Because external module resolution in mylib is independent of that of myotherlib, both will get their respective types.
Sure, it may still be 'hard' to e.g. directly refer to the myutils.SomeOldInterface type from within e.g. myprogram, because we don't have a direct name for it, so probably myutils should simply have 're-exposed' the type for convenience, but e.g. typeof stuff will work, and I can assign the result to any other 'compatible' interface.

[basarat]

In my mail I saw a different post than the one above, seems you've editted it :)

many times :)

[basarat]

Yes my hacky solution would fail miserably for that.

[basarat]

I've made your example concrete: https://github.com/basarat/typescript-node/tree/master/poelstra1/ts

Here is the directory tree :

If you open utils@1 / utils@2 / mylib / myotherlib will just work. The issue does appear at myprogram.ts. Would be great if we could come up with a spec

[poelstra]

@basarat You are (again) amazing 👍 🌹

Note though, that I think the structure you have created now resembles #2338 more than it does this issue (because myutils is a 'native' typescript package).

This is issue is more about the case when myutils is not a typescript package, and its typings are given in a typings/ folder in both mylib and myotherlib.

If you like, I can update your repo with two other folders (maybe clone them to poelstra2 and 3, to allow easy comparison):

• poelstra2 would reflect the 'unwrapped native module' case (i.e. a direct External module resolution logic #2338 test case)
• poelstra3 would reflect the 'unwrapped non-native module' case (i.e. my proposal for this issue)

[poelstra]

Alright, I went ahead, and created those two branches in basarat/typescript-node@22d703a

Obviously, they don't work yet.

• poelstra1 probably never will,
• poelstra2 needs External module resolution logic #2338,
• poelstra3 needs both External module resolution logic #2338 and something along the lines of this thread.

As I had to hand-edit the files/structure and have no way of verifying, there may be e.g. a typo in a path here and there, but I think it conveys the general idea pretty well.

[mhfrantz]

Can you extend the example (which I think is a good idea) to cover the case where the types come from DefinitelyTyped that has not yet been modified as you suggest?

One of the "features" of DefinitelyTyped today is the ability to have a single d.ts file that contains the declarations for many modules. The prime example is node.d.ts. How would that be handled?

Not all of the types are referenced with the appropriate module scoping. For example, in bluebird.d.ts, you can reference it and thus gain access to the Promise class. You can then use that type in your own d.ts file. I suppose the "right" thing to do would be to require the Bluebird module to be imported, and then only refer to the Promise class via the imported name.

I like the theory behind restructuring DefinitelyTyped as outlined here, but I wonder how long it would take to accomplish. I sometimes wish for a parallel repository like npm where we can simply publish the d.ts files by claiming the namespace for individual modules, rather than centrally managing it like the DefinitelyTyped repo.

Alternately, I want to be able to point to one or more alternate typings repos. If your proposal would allow extensibility in the list of repos, then that would allow new-style repos to be developed piecemeal.

[poelstra]

@mhfrantz You make some very interesting remarks!

• extending example with non-modified DT typings: good idea, I'll see what I can come up with
• many declarations in one file:
  • node.d.ts: this is a 'special' case, as Node's docs state that e.g. http always returns its built-in module. So I guess this is one of the few cases where declare module "..." {} still makes sense.
  • all other cases are probably possible once Generate a single declaration file for external modules #2568 is done
• bluebird.d.ts and 'directly' referencing Promise: actually, the way that typing is structured today, is causing us trouble every once in a while; I can call Promise.resolve() in any .ts file in the project, even if I forgot to import Promise = require("bluebird");, it compiles fine, but explodes at runtime. Of course, we could be using import bluebird = require("bluebird"), and use bluebird.resolve(), but we feel that is much less clear. So for us, not having that implicit Promise internal module would be a good thing. May be useful for the browser, but less so for the CommonJS (incl browserify) case, imho.
• restructuring DT:
  • it is not a requirement to restructure DT for this proposal to work, although you'd need to strip the declare module "..." { } off those typings 'by hand' (like we currently have to add it manually to some of them)
  • the existing typings would still work as well (I think, updated example should prove that later)
  • having 'decentralized' typings helps for some cases, but I like the 'quality constraints' that DT enforces before accepting typings. You'd probably get a large soup of low-quality typings if you let that go.
• alternate typings repos: my proposal enforces nothing about which typings repos to use. Basically, I assume that 'somehow' typings end up in a typings/ folder, with a certain naming convention. This 'somehow' can be through tsd (which already allows specifying other repos), you can manually place them there, create your own tsd-variant, etc.

Maybe, for the sake of reducing the scope of this issue, I shouldn't even have mentioned the suggestion to restructure DT. It might be that it happens automatically anyway, once #2338 gets traction.