70 Quotes by Nick Bostrom

The computer scientist Donald Knuth was struck that “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’ – that, somehow, is much harder!

Even with task variety and regular holidays, it is not certain that a human-like mind could live for thousands of subjective years without developing psychological problems.

Solving the value-loading problem is a research challenge worthy of some of the next generation’s best mathematical talent.

Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred.

The orthogonality thesis Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.

Some technical problems in the field of artificial intelligence, for instance, might be negative-value inasmuch as their solution would speed the development of machine intelligence without doing as much to expedite the development of control methods that could render the machine intelligence revolution survivable and beneficial.

Human working memory is able to hold no more than some four or five chunks of information at any given time.

Some little idiot is bound to press the ignite button just to see what happens.

We can think of wisdom as the ability to get the important things approximately right.