The Project DARVIN

In the first half of the 1980s, Hans-Hellmut Nagel began researching driver assistance systems in motor vehicles. This work led to the DARVIN project in 1995, which was implemented at the Fraunhofer-Institut für Informations- und Datenverarbeitung (IITB). The project was accompanied by a website, which has been documented on the Internet Archive since 2003. Originally, this website was available at the Institut für Algorithmen und Kognitive Systeme der Fakultät für Informatik der Universität Karlsruhe. Today, this website can only be accessed via the Internet Archive (as of 22.03.2023). On archive.org you can see how the website has changed over the period from 28.08.2003 to 22.03.2023. However, videos originally available there cannot be viewed. For this reason, Florian Müller and Jakob Tschandl have reconstructed this website for the Deutsches Museum on behalf of the Cryptography and Security Group of the Institute of Information Security and Dependability at KIT. Here you can find information on the DARVIN project including an introduction by Hans-Hellmut Nagel.

The DARVIN website illustrates a special, late intermediate stage in the development of model-based image evaluation of road traffic scenes that has been going on for several decades.

1. Terminology Used

The term image "evaluation" differs significantly from the term image "processing". Image processing generates another digital image from a digital image. Image evaluation, on the other hand, generates a conceptual description of (a part of) the facts captured by the image in the recorded scene from a digital image (see the introductory remarks on the Image Sequences website).

The Image Sequences website documents various road traffic scenes recorded by stationary video cameras of moving road vehicles.

The DARVIN website illustrates various image sequences with superimposed evaluation results, which were recorded from a specially equipped, moving motor vehicle using two video cameras of stationary street scenes.

The use of a priori knowledge about the captured scene makes it possible to skip seemingly insignificant details in the image elements of a digitized video recording and thus avoid computational effort. In view of the limited computing capacities available for analyzing more complex real-world scenes until the 1990s, this aspect could not be ignored. The computing capacity available in a vehicle was even more limited than that of a (large) computing system in a computer center.

Evaluation processes are called "model-based" if they explicitly provide information about scenes whose images are to be evaluated. Explicit information is provided separately from the evaluation commands. This information can therefore be checked, modified or extended without inspecting or changing the sequence of evaluation commands itself.

The explicitly provided information contains the a priori knowledge about the recorded scene to be used for selected (but not too narrowly defined) investigations. Such knowledge, simplified by omission - i.e. abstracted from the complexity of the scene - is also referred to as a "model". For example, a model railroad car does not generally include every detail of a particular railroad car, e.g. a worn spot in a seat upholstery.

The term model is also used for non-geometric aspects of a scene.

2. The History of DARVIN's Image Sequences

The considerations outlined in the above explanation of the terminology prompted H.-H. Nagel in the first third of the 1980s (unfortunately, the documents currently still available do not allow to specify the date) to propose the basic research project outlined below to Prof. Bolle, then Head of Corporate Research at Bosch. At the time, Prof. Bolle was a member of the scientific advisory board of the Fraunhofer-Institut für Informations- und Datenverarbeitung (IITB) in Karlsruhe.

The driver of a motor vehicle should be supported by warnings - if necessary also by automatic, direct interventions in the vehicle control - to recognize critical developments in the environment of the own vehicle at an early stage and to reduce the consequences of a possibly unavoidable traffic accident. (Such an approach can be seen as a visual-system-supported variant of a development that has led, among other things, to the automatic belt tensioner or the ABS anti-lock braking system).

For this purpose, a motor vehicle should be specially equipped for driving maneuvers in city centers in such a way that the knowledge stored in a navigation system could be used to evaluate video image sequences recorded by video cameras behind the windshield of a yet to be developed test vehicle.

Prof. Bolle commissioned the IITB to implement this proposal. In order to achieve initial results as quickly as possible, we wanted to shorten the development time for a suitable test vehicle. The following approach suggested itself.

Prof. E.D. Dickmanns from the University of the Bundeswehr Munich had hired a company to convert a van for automatic driving tests, including a separate diesel generator to produce electricity for the special test electronics required. Prof. Dickmanns kindly allowed the company he had commissioned to convert another van for us.

This gave the IITB a test vehicle that was subsequently adapted to our needs and experience in the IITB workshop and by members of the research group - the green van in the illustration.

Together with the development order from Bosch, we received a file from this company with the navigation system data of a small map section around the IITB site in Karlsruhe. On this basis, the research group at IITB developed an approach to use the data from a navigation system for automatic driving tests, initially on the IITB site in Karlsruhe.

After convincing demonstrations of the capabilities acquired in the meantime, the city of Karlsruhe also permitted driving tests in the immediate vicinity of the IITB, provided that a driver was sitting at the steering wheel ready to intervene and to take immediate control of the test vehicle in critical situations: this was designed in such a way that the automatic driving mode was immediately switched off when the driver intervened. Control of the test vehicle, including the accelerator and brake pedal, was transferred to the driver with virtually no delay.

In the meantime, Prof. Bolle had retired. His successor was Dr. Siegle, who continued these efforts. Under his direction, an extensive demonstration project was planned and carried out in the city center of Hildesheim, at the headquarters of the Bosch laboratory responsible for our project. In coordination with the Hildesheim city administration, the roads had to be closed, while our demonstration vehicle drove there.

The results of this demonstration test were presented at the Intelligent Vehicles Symposium in 1992 (G. Siegle, J. Geisler, F. Laubenstein, H.-H. Nagel, and G. Struck: "Autonomous Driving on a Road Network". In: "Proc. Intelligent Vehicles Symposium IVS'92, 29 June-1 July 1992, Detroit/MI", pp. 403-408).

At about the same time, a BMW Touring was converted at the IITB after suitable preparation by the BMW company such that vision-system-supported driving tests could also be carried out with a passenger car (see the dark BMW with the number KA LR270 in the picture above). This "second generation test vehicle" (see picture above) was equipped with - for the time - powerful special computers, some of which had been developed at the IITB. During the conversion, special attention was paid to ensure that only the two video cameras behind the windshield were visible in the passenger compartment, but no other electronic components or cable connections: the passengers should not be distracted from their driving pleasure by visible electronic components. This test car, which was demonstrated to BMW in Munich, was also used to drive fully automatically on the highway between Karlsruhe and Munich with the aid of a vision system. A member of the research group sat behind the steering wheel, ready to intervene, without touching it (apart from a few critical situations) during the vision-supported drive, as required by the conditional driving permit.

This BMW Touring was used at the IITB for experiments on, among other things, the design of driver-vehicle interaction in sensor-assisted driving.

3. DARVIN test vehicle

From 1995 onwards, another passenger car - a VW Passat Variant GL (1996) (see picture above) - was converted at the IITB for driving tests based on data from a navigation system.

The focus of the research work was on calculating the road geometry in real time from the data stored in a navigation system. This made it unnecessary to carry explicit models of the road to be traveled on. Details of this approach can be found on the DARVIN website below.

The final results achieved with this test vehicle DARVIN were published: F. Heimes and H.-H. Nagel: "Towards Active Machine-Vision-Based Driver Assistance for Urban Areas", International Journal of Computer Vision 50:1 (2002) 5-34.

4. Concluding Remarks

In 2000, H.-H. Nagel retired from the IITB. Responsibility for the test vehicles and for further tests passed into other hands.

Hans-Hellmut Nagel, KIT, März 2024

DARVIN is a short term for

Driver
Assistance using
Realtime
Vision for
INnercity areas

Aplications

DARVIN represents the third generation experimental platform for machine-vision-based road vehicle control. Its equipment has been designed for the development and experimental verification of algorithmic driver assistance in urban traffic. The project is funded by the Fraunhofer-Gesellschaft and managed by H.-H. Nagel.

• Model based intersection tracking
• Model based vehicle tracking
• Traffic situation analysis and conceptual description
• Driver assistance by spoken recommendations and supporting vehicle control
• Support for handicapped drivers

Computer Control of

• Hydraulic steering system
• Brake system
• Throttle system
• Gear change system
• Light, blink signal control

These systems can be operated by the computer and manual operation by the driver can be detected.

Equipment

• Vehicle:
  Volkswagen Passat Variant GL (1996)
• Computers:
  2 x Doppel-PentiumII-300Mhz
  2 x PentiumI-233Mhz
  Flat TFT Monitor
  Fast Ethernet coupled
• Stereo CCD-Camera-Setup
• Steering angle sensor
• GMR (magneto-resistive) wheel sensors

The equipment was mainly installed by Haag Rehatechnik, Östringen. The GMR sensors were kindly donated by Robert Bosch GmbH.

Development Goal

A driver assistance system is developed which emphasizes in particular the boundary conditions for (partially) disabled drivers. It is our goal to study how driver assistance can benefit from a combination of image sequence evaluation and logic interpretation of intermediate results on the basis of a conceptual representation. Based on a better understanding of the current traffic situation gained by this process, the system is expected to ease the driving task for the driver, as long as he does not overrule the system; especially lane and distance keeping may be automated in the future. It is expected that the complexity of situations which an assistance system has to handle will increase continuously in the future. A fully autonomous system - a.k.a. autopilot - has to cope with any situations what so ever. An autopilot would have to operate far more reliably than an average human driver. It can not yet be estimated whether and when such a system can be realized.

Status Quo

A new sedan has been fully equipped with computers, cameras, sensors, computer controlled actuators for accessing steering, gas pedal, brake pedal, gear switch, light, and direction indicators. The software constituting the driver assistance system is currently being developed. The system can already determine its relative ego-position by means of model based image sequence evaluation. It can drive the car along a given path without feedback from the vision system, i. e. the controller and image evaluation system are not yet connected.

Availability

Our current work must be considered as a basic research project. The costs and efforts required to take (parts of) our system to market can not yet be estimated. It is understood that the costs of such systems have to be brought down to the price level of current navigation systems. Our results are published in relevant journals and conferences.

Difference to Other Autonomous Systems

In this project, we focus on inner-city traffic situations since these offer new challenges compared to motorways and motorway-like roads, regarding in particular a wide variation in the kind, speed, and heading of other road users, dense buildings and complex, partially occluded intersections.

This complexity asks for an approach that can cope reliably with a variety of different situations, instead of a quick fix for a specific task. For this reason, we are studying an approach based on model based image sequence evaluation.

Use for Disabled Drivers

Disabled drivers need a driving license as all drivers do. It is expected that a driver assistance system may offer more self confidence to disabled drivers. The first systems can therefore be more specialized and more expensive than a system designed for the normal driver.

Results

Results on vision-based driver assistance based on automatically generated intersection models have been presented at the Intelligent Vehicles Symposium on 4 October 2000 in Dearborn/MI.

Several experiments were conducted under real-time conditions. The vehicle ego-motion was estimated by fitting a scene domain model to binocular camera images. Please refer to the conference proceedings for further details.

The following maneuvers were performed autonomously by the experimental vehicle:

You can download and view the presentation. Run or unzip folien.exe (46 MB selbstextrahierend) and follow the instructions in the file README.TXT

• F. Heimes, K. Fleischer, H.-H. Nagel: "Automatic Generation of Intersection Models from Digital Maps for Vision-Based Driving on Innercity Intersections." In: "Proc. IEEE Intelligent Vehicles Symposium, 3. - 5. Oktober 2000, Dearborn/MI", pp. 498 - 503.
• F. Heimes: "Sichtsystemgestützte Fahrerassistenz im innerstädtischen Straßenverkehr." Dissertation, computer science department, university of Karlsruhe, July 2000; published in "Dissertationen zur Künstlichen Intelligenz (DISKI)" 243, Infix-Verlag, St. Augustin, 2000, ISBN 3-89838-243-5.
• H.-H. Nagel, F. Heimes, K. Fleischer, M. Haag, H. Leuck, S. Noltemeier: "Quantitative Comparison between Trajectory Estimates Obtained from a Binocular Camera Setup within a Moving Road Vehicle and from the Outside by a Stationary Monocular Camera." Special Issue: Applications of Computer Vision to Intelligent Vehicles; Image and Vision Computing 18:5 (2000) 435 - 444.
• H.-H. Nagel, F. Heimes, S. Noltemeier, M. Haag: "Quantitative Comparison between Trajectory Estimates Obtained from a Binocular Camera Setup within a Moving Road Vehicle and from the Outside by a Stationary Monocular Camera." In: "Proc. Intelligent Vehicles Symposium, 28. - 30. Oktober 1998, Stuttgart", pp. 423 - 429.
• F. Heimes, H.-H. Nagel: "Real-Time Tracking of Intersections in Image Sequences of a Moving Camera." In: A. Broggi (Ed.): "Special Issue on Machine Vision for Intelligent Vehicles and Autonomous Robots"; Int. Journal of Engineering Applications of Artificial Intelligence (EAAI), 11 (1998) 215 - 227.
• F. Heimes, H.-H. Nagel, T. Frank: "Model-Based Tracking of Complex Innercity Road Intersections." In: Amin et al. (Eds.): "Special Issue on Intelligent Transportation Systems - Traffic Sensing and Management"; Mathematical and Computer Modelling 27:9 - 11 (1998) 189 - 203.
• M. Tonko, K. Schäfer, F. Heimes, H.-H. Nagel: "Towards Visually Servoed Manipulation of Car Engine Parts." In: R. W. Harrigan (Ed.): "Proc. IEEE Int. Conf. on Robotics and Automation, Albuquerque/NM, 20. - 25. April 1997", IEEE Computer Society Press, Los Alamitos/CA, 1997, pp. 3166 - 3171.
• F. Heimes: "Sichtsystemgestützte Erkennung von Straßenverkehrsszenen." Diploma thesis, Institut für Algorithmen und Kognitive Systeme, computer science department, university of Karlsruhe, August 1995; carried out at the Fraunhofer-Institut für Informations- und Datenverarbeitung (IITB), Karlsruhe.
• V. Gengenbach, H.-H. Nagel, F. Heimes, G. Struck, H. Kollnig: "Model-Based Recognition of Intersections and Lane Structures." In: "Proc. Intelligent Vehicles ´95 Symposium, 25. - 26. September 1995, Detroit/MI", pp. 512 - 517.

We also created a website for the image sequences by Hans-Hellmut Nagel.