US20020163548A1 - System and apparatus for notetaking with digital video and ink - Google Patents
System and apparatus for notetaking with digital video and ink Download PDFInfo
- Publication number
- US20020163548A1 US20020163548A1 US10/185,309 US18530902A US2002163548A1 US 20020163548 A1 US20020163548 A1 US 20020163548A1 US 18530902 A US18530902 A US 18530902A US 2002163548 A1 US2002163548 A1 US 2002163548A1
- Authority
- US
- United States
- Prior art keywords
- notetaking
- media
- input
- video
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 claims description 20
- 230000008859 change Effects 0.000 claims description 17
- 230000007246 mechanism Effects 0.000 claims description 8
- 238000009826 distribution Methods 0.000 claims description 5
- 239000003086 colorant Substances 0.000 claims 1
- 239000000463 material Substances 0.000 description 11
- 230000000694 effects Effects 0.000 description 9
- 238000001514 detection method Methods 0.000 description 8
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000002360 preparation method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 235000005979 Citrus limon Nutrition 0.000 description 1
- 244000131522 Citrus pyriformis Species 0.000 description 1
- 241001310793 Podium Species 0.000 description 1
- 241001255741 Vanna Species 0.000 description 1
- 239000013065 commercial product Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B5/00—Electrically-operated educational appliances
- G09B5/06—Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/171—Editing, e.g. inserting or deleting by use of digital ink
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
- G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/34—Indicating arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/482—End-user interface for program selection
- H04N21/4828—End-user interface for program selection for searching program descriptors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8455—Structuring of content, e.g. decomposing content into time segments involving pointers to the content, e.g. pointers to the I-frames of the video stream
Definitions
- This invention relates to an integrated multimedia notetaking system.
- the invention is more particularly related to a notetaking system that utilizes digital video and ink as references and notes.
- the invention is further related to a notetaking system utilizing video feeds for provision of illustrative notes, book marking, and indexing material.
- the invention is still further related to the indexing of at least one of notes and a video feed via the use of thumbnails, timestamps, and background snaps.
- the invention is yet further related to a notetaking system having a slide detection process for automatic notetaking, and as a feed mechanism for frame rate compression for optimizing bandwidth when presenting material to the notetaking system.
- Multimedia notetaking systems typically capture audio and video during a meeting and slides are created from the captured material.
- Tivoli a system running on LiveBoard, see Moran, T. P., Palen, L., Harrison, S., Chiu, P., Kimber, D., Minneman, S., van Melle, W., and Zellweger, P. “I'll get that off the audio”: a case study of salvaging multimedia meeting records. Proceedings of CHI ′97 CM, New York, pp. 202-209).
- Tivoli is designed to support working meetings rather than presentation meetings.
- the ink strokes in Tivoli, which are indexed to the audio, along with any prepared material on the Tivoli slides become the group notes to the meeting.
- a participant using a laptop may “beam” typed text comments onto a slide in Tivoli.
- classroom 2000 images of presentation slides and audio are captured, but video is not used (see Abowd, G. D., Atkeson, C. G., Brotherton, J., Enqvist, T., Gulley, P., and LeMon, J. Investigating the capture, integration and access problem of ubiquitous computing in an educational setting. Proceedings of the CHI ′98 Conference. ACM, New York, pp. 440-447; and Abowd, G. D., Atkeson, C. G., Feinstein, A., Hmelo, C., Kooper, R., Long, S., Sawhney, N., and Tani, M. Teaching and learning as multimedia authoring: the classroom 2000 project.
- the Forum see Isaacs, E. A., Morris, T., and Rodriguez, T. K. A forum for supporting interactive presentations to distributed audiences. Proceedings of CSCW ′94. ACM, New York, pp. 405-416
- CSCW ′94. ACM New York, pp. 405-416
- Slides have to be prepared in a specified format. The slides can be annotated with text and marks drawn with a mouse, but the video images cannot be annotated.
- the STREAMS (see Cruz, G., and Hill, R. Capturing and playing multimedia events with STREAMS. Proceedings of the ACM Multimedia ′94 Conference. ACM, New York, pp. 193-200), is a system for presentation capture that uses video from room cameras. These cameras are also used to capture any presentation content on display. This method has problems when activity in the room obscures the display. Note-taking during the presentation is not supported, although the captured video streams can be annotated during review by adding text comments. None of these systems allow interactive integration of live images from cameras and presentation material into the notes.
- ACM New York, pp. 271-276
- a PC computer with a pen tablet in which audio is indexed with handwritten notes
- NoTime see Lamming, M., and Newman, W. Activity-based information technology in support of personal memory.
- Technical Report EPC-1991-103, Rank Xerox, EuroPARC, 1991) was designed to key the user's ink strokes to recorded audio or video.
- Marquee see Weber, K., and Poon, A. Marquee: a tool for realtime video logging. Proceedings of CHI ′94. ACM, New York, pp.58-64) is a pen-based system for making annotations while watching a videotape.
- a later version of Marquee has modifications to take timestamps on digital video streams from the WhereWereWe multimedia system (see Minneman, S., Harrison, S., Janssen, B., Kurtenbach, G., Moran, T., Smith, I., and van Melle, B. A confederation of tools for capturing and accessing collaborative activity. Proceedings of the ACM Multimedia ′95 Conference. ACM, New York, pp.523-534).
- Vanna see Harrison, B., Baecker, R. M. Designing video annotation and analysis systems, Graphics Interface ′92. Morgan-Kaufmann, pp. 157-166
- EVA see MacKay, W. E. EVA: An experimental video annotator for symbolic analysis of video data, SIGCHIBulletin, 21 (2), 68-71. 1989. ACM Press
- VideoNoter Trigg, R. Computer support for transcribing recorded activity, SIGCHI Bulletin, 2 1 (2), 68-71. 1989. ACM Press
- displays and synchronizes different streams of activity video, figures whiteboard drawings, text), but requires post-production to transcribe text from the audio or extract drawings from a whiteboard.
- Sharp Zaurus Zaaurus Operation Manual. Sharp Corporation, 1996) is a commercial product, which is a PDA with a digital camera attached. Digital photos can be taken and linked to handwritten notes.
- the present inventors have realized that notetaking is a common activity that can be made more powerful with video.
- the present inventors have also realized the need to provide a fully integrated digital video and ink notetaking system.
- the invention includes a notetaking device that includes, a media input mechanism configured to retrieve at least one media stream, at least one user input mechanism configured to accept user inputs, and a control device configured to allow the user to at least one of manipulate, connote, and summarize the at least one media stream via the user inputs.
- NoteLook includes a client application that runs on a pen-based notebook computer.
- NoteLook has a display with a main area resembling a paper notebook page for writing, capturing, and annotating images.
- the user may change channels to view different video streams.
- the user can grab a frame that is showing in the video window as a small thumbnail in the margin of a note page or as a large background.
- the thumbnails, background images, and ink strokes are time stamped and provide indexes into the video.
- the video source is handled by a NoteLook server, which runs on a computer that has the video input.
- the NoteLook server also transmits the video, audio, as well as meta data (times of slide changes, or speaker changes, for example) to the NoteLook client application, typically via a wireless or wired network.
- These streams of multimedia data are archived by the NoteLook server and can be randomly accessed by the clients during playback. Multiple instances of NoteLook clients and servers can operate together.
- the video source can be captured in a variety of ways: from a room camera or document camera, from a tap into a rear projector, TV, VCR or any video stream, or from a small portable camera attached to a pen computer.
- the NoteLook digital video and ink note-taking system can be used to snap still images of the speaker, room activity, and presentation material and integrate them into the notes.
- the snapped images and ink strokes can be timestamped and linked to the recorded video for easy browsing and retrieval.
- Video can capture gestures, nonverbal activity, and show context. Video provides a versatile means of capturing the presentation content in a variety of forms. PowerPoint slides, Web pages, overhead slides, whiteboard, and more dynamic media such as animation and video can all be captured with video
- FIG. 1 is a picture of a NoteLook client running on a pen based computer
- FIG. 2A is a forward looking view of an FXPAL conference room
- FIG. 2B is a reverse view of the FXPAL conference room
- FIG. 3 is a screen shot of the NoteLook client application
- FIG. 4 is a closeup of video controls and timeline display indexes of the NoteLook client
- FIG. 5 is a block diagram illustrating one possible embodiment of a notetaking device according to the present invention.
- FIG. 6 is a block diagram illustrating components of a processor for the notetaking device according to the present invention.
- FIG. 7 is an example of one possible embodiment of a note file produced by the notetaking system according to the present invention.
- FIG. 8 is a block diagram illustrating an architecture for the NoteLook system.
- FIG. 1 there is illustrated a pen based computer 100 running a NoteLook client application 110 .
- a pen-based notebook computer especially when used with a wireless network, has a form factor that is unobtrusive for note-takers and other participants in the room.
- the NoteLook client is built on top of the FXPAL Dynomite ink and audio notetaking application. It has a main area resembling a paper notebook page for writing, capturing and annotating images. There is a small video window for viewing the active video The user may change channels to view different video streams, grab a frame that is showing in the video window as a small thumbnail in the margin of a note page, or grab a frame as a large background image. The thumbnails, background images, and ink strokes are timestamped and provide indexes into the video.
- the video source is handled by the NoteLook Server, which runs on a computer that has the video input.
- the NoteLook Server also transmits the video, audio, as well as meta data to the NoteLook client application via a wireless (or wired) network.
- Multiple instances of NoteLook Servers and clients can operate together, even on the same machine.
- a server is configured for each video channel, for example, three video channels would be handled by three separate servers, which would service any number of notebook clients.
- the video source can be captured in a variety of ways: from a room camera or document camera, from a tap into a rear projector, TV, VCR or any video stream, or from a small portable camera attached to a pen computer.
- skewing is a problem with employing an externally mounted camera to capture presentation content, and, in at least one embodiment, have provided full digital capture for graphics and other presentation material.
- the NoteLook server has been implemented and integrated with the Kumo conference room at FXPAL, see FIGS. 2A and 2B.
- the rear projector 210 provides another channel for the presentation content, which comes from a computer, document camera, and VCR/DVD (not shown). This set up requires minimal preparation from the speaker.
- the speaker can plug in her own laptop with PowerPoint slides or Web pages, feed overhead slides to the document camera, or play videotapes in the VCR. None of this requires much more extra preparation beyond normal work practice.
- An advanced feature of the NoteLook system is automatic note-taking using slide change detection.
- the video from the rear projector containing the presentation material is analyzed to determine when slide changes occur. Individual slides are extracted and a set of note pages containing these slide images are produced. For indexing video, the points in time when slide changes occur are displayed by marks on the timeline to give visual information that facilitates browsing.
- a further application of slide change detection is frame rate compression for optimizing bandwidth over the wireless network.
- the digital video is captured at 15 or 30 frames a second by the NoteLook server and stored for playback. Since slides do not change anywhere near that rate, a presenter might change a slide every minute or two, it makes sense for the NoteLook server to transmit a video frame only when there is a slide change.
- NoteLook 2GO is a completely portable ink and video note-taking system. It takes the basic NoteLook client running on a pen-based notebook computer and adds a small video camera. This can be used for doing trip reports or field studies.
- NoteLook 4TV is the basic NoteLook client configured with video input from a TV or VCR. This can be used for distance learning and analysis of videotapes or broadcast video.
- each of the above embodiments are clearly applicable to other situations, for example, either may be utilized effectively in a classroom setting.
- the NoteLook client is an ink and audio notetaking application that has video window and video handling capabilities.
- Current development work has been performed on Windows 95/NT, although the features and capabilities described herein are applicable and may be developed on any operating system of ordinary capability.
- a screenshot 300 of NoteLook is shown in FIG. 3, including a close-up of the video window and video controls 400 is shown in FIG. 4.
- the top button channel changer button 405
- the middle button 410 snaps a thumbnail into the margin of the note page
- the bottom button 415 snaps in a large background image (i.e., an image available for annotation or perhaps enlarged for viewing).
- a timeline 425 has a pointer for indicating the current time position of the video.
- the timeline can also display indexing information to facilitate browsing and reviewing.
- FIG. 4 also shows a close-up of the timeline generated from the session shown in FIG. 3.
- a thumbnail 310 , for example
- a background a tiny version of that image is placed in the timeline.
- a band with same color as the pen color is placed in timeline.
- Automatically computed meta data is also displayed; for example, a dotted line 440 in FIG. 4 indicates that a slide change has been detected at that point.
- a video window 320 on the top left corner allows the user to view the active video.
- FIG. 5 illustrates a block diagram of one possible implementation of a device for operating the NoteLook client 110 .
- Media stream(s) and Meta Data 510 are received from the NoteLook server.
- a storage device 550 maintains storage of the NoteLook program and any note files or other data created or saved during a NoteLook session.
- a channel selector 530 directs an appropriate stream to display 560 (for display in video window 320 ).
- a processor 540 receives user inputs (such as channel selection) and directs channel selection, and output (button placement, etc) to the display 560 .
- the processor 540 may also initiate user requests sent to the server (recall or playback of a video at a specific timestamp or snapshot, for example).
- the processor 540 When loaded with the NoteLook program, the processor 540 performs the operations described above.
- the processor itself may be illustrated, but not limited to, specific devices as those shown in the block diagram of FIG. 6.
- each instance of a NoteLook client produces a note file consisting of digital ink strokes, thumbnails, and background snaps. These objects are timestamped and attributed with a channel number, which provides indexes into the video streams.
- FIG. 7 illustrates one possible implementation of a note file according to the present invention.
- the note file references any one of the objects or items 700 input by the NoteLook 100 (thumbnail 710 , ink strokes 715 , and snap 720 , for example). Each object has a corresponding timestamp 750 and channel identifier 770 . Individual note files and additional information may also be maintained.
- a user can review the notes and play the video corresponding to an object by selecting it and pressing the PLAY button. Clicking or double clicking on the thumbnails is another way to play the video. Playing an object will start the video at the (time, channel) corresponding to (time, channel) when the object was created (alternatively the thumbnails may point or link to specific video or media segment).
- the user may change channels during playback by pressing the channel changer button ( 405 ).
- the timeline which visually maps out the note-taking activity by time, provides yet another way to browse and play the video at points of interest. To play, the user simply drags the pointer on the timeline to the desired place or clicks at a location on the timeline.
- the digital video streams are saved separately by the server onto the network, and can be randomly accessed via note files taken by different individuals.
- the video can be saved locally with the note files, but this requires a large amount of file space.
- the NoteLook client runs on a pen-based notebook computer: Fujitsu Stylistic 1200 (Pentium 120, 8 inch VGA active matrix display).
- Fujitsu Stylistic 1200 Patentium 120, 8 inch VGA active matrix display.
- a WaveLan PC Card (10 Mb) is employed.
- the video is transmitted to the NoteLook clients by the NoteLook server over a wireless network 810 (see FIG. 8.)
- the NoteLook server runs on Windows 95 or Windows NT (other operating systems are clearly applicable).
- the server machine is equipped with video capture cards, and accepts composite video.
- the digital video is captured at 15 or 30 frames a second and stored onto the network for playback.
- the main video sources are three room cameras ( 200 , 201 , and 202 ) and a 120-inch rear projector 210 .
- FIG. 5 shows one embodiment of the architecture of NoteLook.
- Video of presentation material is fed into the rear projector from a workstation 820 in the back room, such as a user-supplied laptop, document camera, VCR, or DVD/CD player, for example.
- the speaker selects'the desired rear projector source at the podium.
- a room camera can be aimed at a whiteboard in the room and show it on the rear projector.
- Audio in Kumo is captured by six ceiling microphones, combined into a single stream of audio, and stored on the network with the video. Many other combinations of audio, video, or other data may be provided as a media stream to NoteLook clients.
- NoteLook 2GO and NoteLook 4TV can be implemented by running the NoteLook client and server on the same pen-based notebook computer.
- NoteLook 2GO a portable camera with a USB connection is currently preferred; the USB port provides power to the camera.
- NoteLook 4TV a PC Card for video capture is currently preferred.
- Frame rate compression may be utilized to save bandwidth.
- Automatic change detection (slide change detection, for example) may be utilized and only changes in a media stream are sent to the NoteLook clients.
- a frame per second rate of material in the media stream may be reduced (i.e., a video shot at 30 fps may be reduced to 1 or 2 fps). This may result in a jittery display at the NoteLook client, however, a perfect video at the notetaking device is not essential to the notetaking process, and, in fact, not needed in situations where the notetaker is present at the video (because the notetaker can hear and see what is taking place)
- Auto indexing may also be utilized to automatically determine important video shots that may automatically be captured and made into specific notes (using the same change detection criteria discussed above). Such auto indexed material may also be linked to the timeline (such as the slide change as discussed above).
- the present invention may be conveniently implemented using a conventional general purpose or a specialized digital computer or microprocessor programmed according to the teachings of the present disclosure, as will be apparent to those skilled in the computer art.
- the present invention includes a computer program product which is a storage medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention
- the storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
- the present invention includes software for controlling both the hardware of the general purpose/specialized computer or microprocessor, and for enabling the computer or microprocessor to interact with a human user or other mechanism utilizing the results of the present invention.
- software may include, but is not limited to, device drivers, operating systems, and user applications.
- computer readable media further includes software for performing the present invention, as described above.
Abstract
Media input streams are retrieved by a notetaking device and referenced for playback. The media streams are from microphones, video cameras, and media players present during a notetaking session. Facilities provided by the notetaking device allow a user to capture stills from the media streams, make annotations, and reference important events that occur during a notetaking session. Thumbnails, snaps, and backgrounds may be created from the input media streams and are used to reference into the media streams which are stored for later playback. A channel changer allows a user to switch between the various media streams during either notetaking or playback. A timeline is automatically generated that references important events, including slide changes, presented during a notetaking session.
Description
- 1. FIELD OF THE INVENTION
- This invention relates to an integrated multimedia notetaking system. The invention is more particularly related to a notetaking system that utilizes digital video and ink as references and notes. The invention is further related to a notetaking system utilizing video feeds for provision of illustrative notes, book marking, and indexing material. The invention is still further related to the indexing of at least one of notes and a video feed via the use of thumbnails, timestamps, and background snaps. The invention is yet further related to a notetaking system having a slide detection process for automatic notetaking, and as a feed mechanism for frame rate compression for optimizing bandwidth when presenting material to the notetaking system.
- 2. DISCUSSION OF THE BACKGROUND
- Multimedia notetaking systems typically capture audio and video during a meeting and slides are created from the captured material. (For example, Tivoli, a system running on LiveBoard, see Moran, T. P., Palen, L., Harrison, S., Chiu, P., Kimber, D., Minneman, S., van Melle, W., and Zellweger, P. “I'll get that off the audio”: a case study of salvaging multimedia meeting records. Proceedings of CHI ′97 CM, New York, pp. 202-209). Tivoli is designed to support working meetings rather than presentation meetings. The ink strokes in Tivoli, which are indexed to the audio, along with any prepared material on the Tivoli slides become the group notes to the meeting. A participant using a laptop may “beam” typed text comments onto a slide in Tivoli.
- In a similar example, Classroom 2000, images of presentation slides and audio are captured, but video is not used (see Abowd, G. D., Atkeson, C. G., Brotherton, J., Enqvist, T., Gulley, P., and LeMon, J. Investigating the capture, integration and access problem of ubiquitous computing in an educational setting. Proceedings of the CHI ′98 Conference. ACM, New York, pp. 440-447; and Abowd, G. D., Atkeson, C. G., Feinstein, A., Hmelo, C., Kooper, R., Long, S., Sawhney, N., and Tani, M. Teaching and learning as multimedia authoring: the classroom 2000 project. Proceedings of the ACM Multimedia ′96 Conference. ACM, New York, pp. 187-198). In addition, Classroom2000 requires effort by the presenter to prepare the slides in a standard graphics format. The slides are displayed on a LiveBoard and note-taking is done with PDA devices pre-loaded with slides. These notes are later synchronized to the audio and the slides which have been annotated by the professor lecturing in front of the LiveBoard.
- In yet another example, the Forum (see Isaacs, E. A., Morris, T., and Rodriguez, T. K. A forum for supporting interactive presentations to distributed audiences. Proceedings of CSCW ′94. ACM, New York, pp. 405-416), is a system uses video as a means for distributed presentations. Everyone, including the speaker, sits in front of a workstation during a presentation. Slides have to be prepared in a specified format. The slides can be annotated with text and marks drawn with a mouse, but the video images cannot be annotated.
- In another example, the STREAMS (see Cruz, G., and Hill, R. Capturing and playing multimedia events with STREAMS. Proceedings of the ACM Multimedia ′94 Conference. ACM, New York, pp. 193-200), is a system for presentation capture that uses video from room cameras. These cameras are also used to capture any presentation content on display. This method has problems when activity in the room obscures the display. Note-taking during the presentation is not supported, although the captured video streams can be annotated during review by adding text comments. None of these systems allow interactive integration of live images from cameras and presentation material into the notes.
- In addition, there are also several known stand alone ink and audio note-taking systems. For example, FXPAL Dynomite (see Wilcox, L. D., Schilit, B. N., and Sawhney, N. Dynomite: A Dynamically Organized Ink and Audio Notebook. Proceedings of CHI ′97. ACM, New York, pp. 186-193); and Audio Notebook (see Stifelman, L. The Audio Notebook: Paper and Pen Interaction with Structured Speech. PhD Thesis. MIT, 1997), which uses paper with audio recording. Filochat (see Whittaker, S., Hyland, P., and Wiley, M. Filochat: handwritten notes provide access to recorded conversations. Proceedings of CHI ′94. ACM, New York, pp. 271-276), is a PC computer with a pen tablet in which audio is indexed with handwritten notes; and NoTime (see Lamming, M., and Newman, W. Activity-based information technology in support of personal memory. Technical Report EPC-1991-103, Rank Xerox, EuroPARC, 1991), was designed to key the user's ink strokes to recorded audio or video.
- Also known are video annotation systems. Marquee (see Weber, K., and Poon, A. Marquee: a tool for realtime video logging. Proceedings of CHI ′94. ACM, New York, pp.58-64) is a pen-based system for making annotations while watching a videotape. A later version of Marquee has modifications to take timestamps on digital video streams from the WhereWereWe multimedia system (see Minneman, S., Harrison, S., Janssen, B., Kurtenbach, G., Moran, T., Smith, I., and van Melle, B. A confederation of tools for capturing and accessing collaborative activity. Proceedings of the ACM Multimedia ′95 Conference. ACM, New York, pp.523-534).
- Vanna (see Harrison, B., Baecker, R. M. Designing video annotation and analysis systems, Graphics Interface ′92. Morgan-Kaufmann, pp. 157-166); and EVA (see MacKay, W. E. EVA: An experimental video annotator for symbolic analysis of video data, SIGCHIBulletin, 21 (2), 68-71. 1989. ACM Press) are text based systems. VideoNoter (Trigg, R. Computer support for transcribing recorded activity, SIGCHI Bulletin,21 (2), 68-71. 1989. ACM Press) displays and synchronizes different streams of activity (video, figures whiteboard drawings, text), but requires post-production to transcribe text from the audio or extract drawings from a whiteboard. These systems are limited by their design based on using videotapes rather than digital video. None of these systems allow interactive integration of video images into the notes. Sharp Zaurus (Zaurus Operation Manual. Sharp Corporation, 1996) is a commercial product, which is a PDA with a digital camera attached. Digital photos can be taken and linked to handwritten notes.
- The present inventors have realized that notetaking is a common activity that can be made more powerful with video. The present inventors have also realized the need to provide a fully integrated digital video and ink notetaking system.
- Accordingly, it is an object of the present invention to provide a multimedia notetaking system.
- It is another object of the present invention to provide a notetaking system that allows the user to annotate images captured from a media stream input to the notetaking system.
- It is yet another object of the present invention to provide a notetaking system having a timeline that identifies significant events occurring during a notetaking session.
- It is yet another object of the present invention to provide a notetaking system that captures live multimedia steams and utilizing framerate compression to provide the multimedia streams to a notetaking device and to automatically summarize events such as slide changes into a timeline.
- And it is still further yet another object of the present invention to allow a user to bookmark points in a captured multimedia stream in a notetaking device.
- These and other objects are accomplished by a system for note-taking with digital video and ink (also referred to as NoteLook). The invention includes a notetaking device that includes, a media input mechanism configured to retrieve at least one media stream, at least one user input mechanism configured to accept user inputs, and a control device configured to allow the user to at least one of manipulate, connote, and summarize the at least one media stream via the user inputs.
- In one embodiment, NoteLook includes a client application that runs on a pen-based notebook computer. NoteLook has a display with a main area resembling a paper notebook page for writing, capturing, and annotating images. There is a small video window for viewing the active video. The user may change channels to view different video streams. The user can grab a frame that is showing in the video window as a small thumbnail in the margin of a note page or as a large background.
- The thumbnails, background images, and ink strokes are time stamped and provide indexes into the video. The video source is handled by a NoteLook server, which runs on a computer that has the video input. The NoteLook server also transmits the video, audio, as well as meta data (times of slide changes, or speaker changes, for example) to the NoteLook client application, typically via a wireless or wired network. These streams of multimedia data are archived by the NoteLook server and can be randomly accessed by the clients during playback. Multiple instances of NoteLook clients and servers can operate together. The video source can be captured in a variety of ways: from a room camera or document camera, from a tap into a rear projector, TV, VCR or any video stream, or from a small portable camera attached to a pen computer.
- In meetings, presentations and classes, the NoteLook digital video and ink note-taking system can be used to snap still images of the speaker, room activity, and presentation material and integrate them into the notes. The snapped images and ink strokes can be timestamped and linked to the recorded video for easy browsing and retrieval. Video can capture gestures, nonverbal activity, and show context. Video provides a versatile means of capturing the presentation content in a variety of forms. PowerPoint slides, Web pages, overhead slides, whiteboard, and more dynamic media such as animation and video can all be captured with video
- Demonstrations during presentations and training sessions are also effectively captured by video. In order for a digital ink and video note-taking system to be usable, it must be unobtrusive for the note-takers and other participants in the room and require minimal preparation from the speaker.
- A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
- FIG. 1 is a picture of a NoteLook client running on a pen based computer;
- FIG. 2A is a forward looking view of an FXPAL conference room;
- FIG. 2B is a reverse view of the FXPAL conference room;
- FIG. 3 is a screen shot of the NoteLook client application;
- FIG. 4 is a closeup of video controls and timeline display indexes of the NoteLook client;
- FIG. 5 is a block diagram illustrating one possible embodiment of a notetaking device according to the present invention;
- FIG. 6 is a block diagram illustrating components of a processor for the notetaking device according to the present invention;
- FIG. 7 is an example of one possible embodiment of a note file produced by the notetaking system according to the present invention; and
- FIG. 8 is a block diagram illustrating an architecture for the NoteLook system.
- Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts, and more particularly to FIG. 1 thereof, there is illustrated a pen based
computer 100 running aNoteLook client application 110. - Although applicable to other platforms, a pen-based notebook computer, especially when used with a wireless network, has a form factor that is unobtrusive for note-takers and other participants in the room.
- In a current embodiment, the NoteLook client is built on top of the FXPAL Dynomite ink and audio notetaking application. It has a main area resembling a paper notebook page for writing, capturing and annotating images. There is a small video window for viewing the active video The user may change channels to view different video streams, grab a frame that is showing in the video window as a small thumbnail in the margin of a note page, or grab a frame as a large background image. The thumbnails, background images, and ink strokes are timestamped and provide indexes into the video.
- The video source is handled by the NoteLook Server, which runs on a computer that has the video input. The NoteLook Server also transmits the video, audio, as well as meta data to the NoteLook client application via a wireless (or wired) network. Multiple instances of NoteLook Servers and clients can operate together, even on the same machine. In one embodiment a server is configured for each video channel, for example, three video channels would be handled by three separate servers, which would service any number of notebook clients. The video source can be captured in a variety of ways: from a room camera or document camera, from a tap into a rear projector, TV, VCR or any video stream, or from a small portable camera attached to a pen computer.
- The present inventors have also realized that skewing is a problem with employing an externally mounted camera to capture presentation content, and, in at least one embodiment, have provided full digital capture for graphics and other presentation material.
- The NoteLook server has been implemented and integrated with the Kumo conference room at FXPAL, see FIGS. 2A and 2B. In one embodiment, there are three room cameras (200, 201, and 202), each providing a channel of video (additional cameras with alternate or remote views may also be supported). The
rear projector 210 provides another channel for the presentation content, which comes from a computer, document camera, and VCR/DVD (not shown). This set up requires minimal preparation from the speaker. The speaker can plug in her own laptop with PowerPoint slides or Web pages, feed overhead slides to the document camera, or play videotapes in the VCR. None of this requires much more extra preparation beyond normal work practice. - An advanced feature of the NoteLook system is automatic note-taking using slide change detection. The video from the rear projector containing the presentation material is analyzed to determine when slide changes occur. Individual slides are extracted and a set of note pages containing these slide images are produced. For indexing video, the points in time when slide changes occur are displayed by marks on the timeline to give visual information that facilitates browsing.
- A further application of slide change detection is frame rate compression for optimizing bandwidth over the wireless network. The digital video is captured at 15 or 30 frames a second by the NoteLook server and stored for playback. Since slides do not change anywhere near that rate, a presenter might change a slide every minute or two, it makes sense for the NoteLook server to transmit a video frame only when there is a slide change.
- We describe two variations of the system: NoteLook 2GO and NoteLook 4TV. NoteLook 2GO is a completely portable ink and video note-taking system. It takes the basic NoteLook client running on a pen-based notebook computer and adds a small video camera. This can be used for doing trip reports or field studies. NoteLook 4TV is the basic NoteLook client configured with video input from a TV or VCR. This can be used for distance learning and analysis of videotapes or broadcast video. As will be appreciated by those of ordinary skill in the art, each of the above embodiments are clearly applicable to other situations, for example, either may be utilized effectively in a classroom setting. Technical Details
- The NoteLook client is an ink and audio notetaking application that has video window and video handling capabilities. Current development work has been performed on Windows 95/NT, although the features and capabilities described herein are applicable and may be developed on any operating system of ordinary capability.
- A
screenshot 300 of NoteLook is shown in FIG. 3, including a close-up of the video window and video controls 400 is shown in FIG. 4. Next to the video window are three buttons for interacting with the video: the top button (channel changer button 405) changes channels, themiddle button 410 snaps a thumbnail into the margin of the note page, and thebottom button 415 snaps in a large background image (i.e., an image available for annotation or perhaps enlarged for viewing). - Further to the left of the video window is a set of VCR type controls420. A
timeline 425 has a pointer for indicating the current time position of the video. The timeline can also display indexing information to facilitate browsing and reviewing. - FIG. 4 also shows a close-up of the timeline generated from the session shown in FIG. 3. When the user snaps in either a thumbnail (310, for example) or a background, a tiny version of that image is placed in the timeline. In addition, when the user writes with the pen, a band with same color as the pen color is placed in timeline.
- Automatically computed meta data is also displayed; for example, a
dotted line 440 in FIG. 4 indicates that a slide change has been detected at that point. Avideo window 320 on the top left corner allows the user to view the active video. - FIG. 5 illustrates a block diagram of one possible implementation of a device for operating the
NoteLook client 110. Media stream(s) andMeta Data 510 are received from the NoteLook server. Astorage device 550 maintains storage of the NoteLook program and any note files or other data created or saved during a NoteLook session. - A
channel selector 530 directs an appropriate stream to display 560 (for display in video window 320). Aprocessor 540 receives user inputs (such as channel selection) and directs channel selection, and output (button placement, etc) to thedisplay 560. Theprocessor 540 may also initiate user requests sent to the server (recall or playback of a video at a specific timestamp or snapshot, for example). - When loaded with the NoteLook program, the
processor 540 performs the operations described above. The processor itself may be illustrated, but not limited to, specific devices as those shown in the block diagram of FIG. 6. - During a note-taking session, each instance of a NoteLook client produces a note file consisting of digital ink strokes, thumbnails, and background snaps. These objects are timestamped and attributed with a channel number, which provides indexes into the video streams.
- FIG. 7 illustrates one possible implementation of a note file according to the present invention. The note file references any one of the objects or
items 700 input by the NoteLook 100 (thumbnail 710, ink strokes 715, and snap 720, for example). Each object has acorresponding timestamp 750 andchannel identifier 770. Individual note files and additional information may also be maintained. - After the session, a user can review the notes and play the video corresponding to an object by selecting it and pressing the PLAY button. Clicking or double clicking on the thumbnails is another way to play the video. Playing an object will start the video at the (time, channel) corresponding to (time, channel) when the object was created (alternatively the thumbnails may point or link to specific video or media segment).
- The user may change channels during playback by pressing the channel changer button (405). The timeline, which visually maps out the note-taking activity by time, provides yet another way to browse and play the video at points of interest. To play, the user simply drags the pointer on the timeline to the desired place or clicks at a location on the timeline.
- The digital video streams are saved separately by the server onto the network, and can be randomly accessed via note files taken by different individuals. Alternatively, the video can be saved locally with the note files, but this requires a large amount of file space.
- In the set up at FXPAL, the NoteLook client runs on a pen-based notebook computer: Fujitsu Stylistic 1200 (Pentium 120, 8 inch VGA active matrix display). For wireless network connection, a WaveLan PC Card (10 Mb) is employed. The video is transmitted to the NoteLook clients by the NoteLook server over a wireless network 810 (see FIG. 8.) The NoteLook server runs on Windows 95 or Windows NT (other operating systems are clearly applicable). The server machine is equipped with video capture cards, and accepts composite video. The digital video is captured at 15 or 30 frames a second and stored onto the network for playback. In the current set up at FXPAL Kumo conference room, the main video sources are three room cameras (200, 201, and 202) and a 120-inch
rear projector 210. FIG. 5 shows one embodiment of the architecture of NoteLook. Video of presentation material is fed into the rear projector from aworkstation 820 in the back room, such as a user-supplied laptop, document camera, VCR, or DVD/CD player, for example. The speaker selects'the desired rear projector source at the podium. Also, a room camera can be aimed at a whiteboard in the room and show it on the rear projector. Audio in Kumo is captured by six ceiling microphones, combined into a single stream of audio, and stored on the network with the video. Many other combinations of audio, video, or other data may be provided as a media stream to NoteLook clients. - Automatic slide change detection is performed by the NoteLook server (although such detection may be performed on any
NoteLook 520, the NoteLook server is a more efficient location for any required processing and subsequent distribution of results). One algorithm to distinguish between video frames is to employ a method of comparing color histograms of successive frames in the video stream to determine when a frame has changed and hence a slide change has occurred Each detected slide change provides a piece of meta data which is sent to the NoteLook client. The client manages indexes created from the meta data and displays marks on the timeline (the dotted line in FIG. 4). For optimizing the wireless bandwidth, frame rate compression is done by dropping frames between slide changes. The video frames and meta data are transmitted to the NoteLook clients using standard TCP/IP methods (again, other methods may be utilized). - NoteLook 2GO and NoteLook 4TV can be implemented by running the NoteLook client and server on the same pen-based notebook computer. For NoteLook 2GO, a portable camera with a USB connection is currently preferred; the USB port provides power to the camera. For NoteLook 4TV, a PC Card for video capture is currently preferred.
- Frame rate compression may be utilized to save bandwidth. Automatic change detection (slide change detection, for example) may be utilized and only changes in a media stream are sent to the NoteLook clients. Alternatively, a frame per second rate of material in the media stream may be reduced (i.e., a video shot at 30 fps may be reduced to 1 or 2 fps). This may result in a jittery display at the NoteLook client, however, a perfect video at the notetaking device is not essential to the notetaking process, and, in fact, not needed in situations where the notetaker is present at the video (because the notetaker can hear and see what is taking place)
- Auto indexing may also be utilized to automatically determine important video shots that may automatically be captured and made into specific notes (using the same change detection criteria discussed above). Such auto indexed material may also be linked to the timeline (such as the slide change as discussed above).
- The present invention may be conveniently implemented using a conventional general purpose or a specialized digital computer or microprocessor programmed according to the teachings of the present disclosure, as will be apparent to those skilled in the computer art.
- Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art. The invention may also be implemented by the preparation of application specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be readily apparent to those skilled in the art.
- The present invention includes a computer program product which is a storage medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
- Stored on any one of the computer readable medium (media), the present invention includes software for controlling both the hardware of the general purpose/specialized computer or microprocessor, and for enabling the computer or microprocessor to interact with a human user or other mechanism utilizing the results of the present invention. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Ultimately, such computer readable media further includes software for performing the present invention, as described above.
- Included in the programming (software) of the general/specialized computer or microprocessor are software modules for implementing the teachings of the present invention, including, but not limited to capturing and annotating media streams, producing a timeline of significant note-taking events, linking still frames to points in or segments of a media stream, recognize any slide changes, production and distribution of meta data describing at least a part of a media stream, and communication of results according to the processes of the present invention.
- Obviously, numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.
Claims (27)
1. A notetaking system, comprising:
a notetaking device that comprises,
a media input mechanism configured to retrieve at least one live media stream,
at least one user input mechanism configured to accept user inputs, and
a control device configured to allow said user to at least one of manipulate, connote, and summarize said at least one media stream via said user inputs.
2. The notetaking system according to claim 1 , wherein said notetaking device further comprises:
a timeline device that identifies and indexes significant events in a timeline that references said at least one media stream.
3. The notetaking system according to claim 1 , wherein said notetaking device further comprises:
a timestamp mechanism configured to timestamp and link actions taken by said user to said at least one media stream.
4. The notetaking device according to claim 1 , wherein said notetaking device further comprises:
a still frame capture device configured to capture still frames from said at least one media stream; and
an indexer configured to link each still frame to one of a segment or a location in said at least one media stream.
5. The notetaking system according to claim 4 , wherein said notetaking device further comprises:
an annotator configured to,
accept user inputs directed toward an annotation of at least one of the captured frames, and
apply the annotation to the frames that the annotation is directed toward.
6. The notetaking system according to Claim 1 ,wherein said notetaking device further comprises:
an ink stroke capture device configured to capture ink strokes on a note page; and
an indexer configured to link each note page to one of,
a segment of said at least one media stream, and
a location in a timetable of events displayed on said notetaking device.
7. The notetaking system according to claim 6 , wherein each note page link is graphically represented in a same color as the ink strokes captured on the corresponding note page.
8. The notetaking device according to claim 1 , wherein said notetaking device further comprises:
further comprising a channel mechanism configured to identify and select at least one channel of the media streams for display by said notetaking device.
9. The notetaking system according to claim 1 , further comprising:
a note server that provides multimedia inputs, each input corresponding to one of the media streams.
10. The notetaking system according to claim 7 , wherein said inkstrokes of each note page are one of plural user selectable colors.
11. A method of notetaking, comprising the steps of:
retrieving at least one input media stream;
accepting user inputs corresponding to actions to be performed on said at least one input media stream; and
performing at least one of manipulating, annotating, and summarizing said at least one input media stream according to said user input actions.
12. The method according to claim 11 , further comprising the step of:
constructing a timeline referencing events associated with said input media stream.
13. The method according to claim 12 , wherein said step of constructing includes the step of linking at least one of annotations, video clips, slide changes, audio clips, still frames, and annotations as events referenced by said timeline.
14. The method according to claim 12 , further comprising the step of:
storing a timestamp identifying each event.
15. The method according to claim 11 , further comprising the steps of:
capturing still frames from said input media stream; and
linking each of said still frames to one of a point and a segment of said input media stream.
16. The method according to claim 11 , further comprising the step of annotating at least one of the captured still frames.
17. The method according to claim 11 , further comprising the step of selecting said at least one input media stream from at least one input channel.
18. The method according to claim 12 , wherein said events include at least one of slide changes and frame changes of at least one of the input media streams.
19. A note server, comprising:
a capture device configured to capture data input from at least one source device; and
a distribution device configured to distribute the captured data to at least one client device.
20. The note server according to claim 19 , further comprising:
a slide change recognition device configured recognize changes in the captured data and inhibit distribution of all but changed data.
21. The note server according to claim 20 , further comprising:
a metadata device configured to build a set of metadata corresponding to captured data;
wherein said metadata is distributed in connection with the corresponding captured data.
22. The note server according to claim 19 , wherein said server is further configured to save each media input and make each media input available for recall by said at least one client.
23. The note server according to claim 19 , further comprising:
an index recognition device configured to recognize an index to a segment or point in one of the captured data and data stored on said at least one source device;
wherein said index recognition device is further configured to initiate a distribution of said segment or the source device stored data beginning at said point to at least one client.
24. A method of server operation, comprising the steps of:
capturing media data from at least one source device; and
distributing the media data captured to at least one client device.
25. The method according to claim 24 , further comprising the step of:
recognizing changes in the media data captured; and
distributing only a changed media data to said at least one client.
26. The method according to claim 24 , further comprising the steps of:
retrieving an index to a segment or point of one of the captured media data and media data stored on said at least one source device; and
distributing the indexed media data to at least one client device.
27. The method according to claim 24 , further comprising the steps of:
constructing a metadata tag corresponding to the media data to be distributed; and
distributing the metadata tag in connection with the corresponding media data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/185,309 US20020163548A1 (en) | 1999-03-24 | 2002-06-27 | System and apparatus for notetaking with digital video and ink |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/275,285 US6452615B1 (en) | 1999-03-24 | 1999-03-24 | System and apparatus for notetaking with digital video and ink |
US10/185,309 US20020163548A1 (en) | 1999-03-24 | 2002-06-27 | System and apparatus for notetaking with digital video and ink |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/275,285 Division US6452615B1 (en) | 1999-03-24 | 1999-03-24 | System and apparatus for notetaking with digital video and ink |
Publications (1)
Publication Number | Publication Date |
---|---|
US20020163548A1 true US20020163548A1 (en) | 2002-11-07 |
Family
ID=23051635
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/275,285 Expired - Lifetime US6452615B1 (en) | 1999-03-24 | 1999-03-24 | System and apparatus for notetaking with digital video and ink |
US10/185,309 Abandoned US20020163548A1 (en) | 1999-03-24 | 2002-06-27 | System and apparatus for notetaking with digital video and ink |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/275,285 Expired - Lifetime US6452615B1 (en) | 1999-03-24 | 1999-03-24 | System and apparatus for notetaking with digital video and ink |
Country Status (2)
Country | Link |
---|---|
US (2) | US6452615B1 (en) |
JP (1) | JP4171157B2 (en) |
Cited By (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020161804A1 (en) * | 2001-04-26 | 2002-10-31 | Patrick Chiu | Internet-based system for multimedia meeting minutes |
US20030117532A1 (en) * | 2001-12-25 | 2003-06-26 | Seiko Epson Corporation | Projector wireless control system and wireless control method |
US20030142200A1 (en) * | 2002-01-29 | 2003-07-31 | Palm, Inc. | System for and method of conferencing with a handheld computer using multiple media types |
WO2004038982A2 (en) * | 2002-10-25 | 2004-05-06 | Disney Enterprises, Inc. | Streaming of digital data to a portable device |
EP1536348A1 (en) * | 2003-11-26 | 2005-06-01 | Ricoh Company, Ltd. | Techniques for integrating note-taking and multimedia information |
GB2408880A (en) * | 2003-12-03 | 2005-06-08 | Safehouse Internat Inc | Observing monitored image data and highlighting incidents on a timeline |
US20050125717A1 (en) * | 2003-10-29 | 2005-06-09 | Tsakhi Segal | System and method for off-line synchronized capturing and reviewing notes and presentations |
US20050122397A1 (en) * | 2003-12-03 | 2005-06-09 | Safehouse International Limited | Recording a sequence of images |
US20050154995A1 (en) * | 2004-01-08 | 2005-07-14 | International Business Machines Corporation | Intelligent agenda object for showing contextual location within a presentation application |
US20050163212A1 (en) * | 2003-03-31 | 2005-07-28 | Safehouse International Limited | Displaying graphical output |
US20050163345A1 (en) * | 2003-12-03 | 2005-07-28 | Safehouse International Limited | Analysing image data |
US20050163346A1 (en) * | 2003-12-03 | 2005-07-28 | Safehouse International Limited | Monitoring an output from a camera |
US20070038935A1 (en) * | 1999-11-17 | 2007-02-15 | Ricoh Company, Ltd. | Techniques for capturing information during multimedia presentations |
US20070081075A1 (en) * | 2002-01-29 | 2007-04-12 | Palm, Inc. | Videoconferencing bandwidth management for a handheld computer system and method |
US20090027400A1 (en) * | 2007-05-29 | 2009-01-29 | Jim Marggraff | Animation of Audio Ink |
US7647555B1 (en) * | 2000-04-13 | 2010-01-12 | Fuji Xerox Co., Ltd. | System and method for video access from notes or summaries |
US7653925B2 (en) | 1999-11-17 | 2010-01-26 | Ricoh Company, Ltd. | Techniques for receiving information during multimedia presentations and communicating the information |
US7945595B1 (en) | 2008-05-12 | 2011-05-17 | Adobe Systems Incorporated | System and method for generating an item list in electronic content |
US20110119588A1 (en) * | 2009-11-17 | 2011-05-19 | Siracusano Jr Louis H | Video storage and retrieval system and method |
US7949633B1 (en) | 2008-05-12 | 2011-05-24 | Adobe Systems Incorporated | Shared edit access of electronic content |
US20110145726A1 (en) * | 2009-12-10 | 2011-06-16 | Hulu Llc | Method and apparatus for navigating a media program via a histogram of popular segments |
US20120281967A1 (en) * | 2003-01-09 | 2012-11-08 | Kaleidescape, Inc. | Bookmarks and Watchpoints for Selection and Presentation of Media Streams |
US8380866B2 (en) | 2009-03-20 | 2013-02-19 | Ricoh Company, Ltd. | Techniques for facilitating annotations |
US8464164B2 (en) * | 2006-01-24 | 2013-06-11 | Simulat, Inc. | System and method to create a collaborative web-based multimedia contextual dialogue |
WO2013109510A1 (en) * | 2012-01-19 | 2013-07-25 | Microsoft Corporation | Usage based synchronization of note-taking application features |
US8706898B2 (en) | 2009-12-10 | 2014-04-22 | Hulu, LLC | Navigating a video, a transcript of a dialog, and a histogram |
US20140143257A1 (en) * | 2002-12-11 | 2014-05-22 | Trio Systems Llc | Annotation system for creating and retrieving media and methods relating to same |
US8805929B2 (en) | 2005-06-20 | 2014-08-12 | Ricoh Company, Ltd. | Event-driven annotation techniques |
US8996621B2 (en) | 2008-05-12 | 2015-03-31 | Adobe Systems Incorporated | Asynchronous comment updates |
JP2015079494A (en) * | 2013-10-15 | 2015-04-23 | シャープ株式会社 | Information processor, information processing method, and recording medium |
US20150116283A1 (en) * | 2013-10-24 | 2015-04-30 | Livescribe Inc. | Paper Strip Presentation Of Grouped Content |
US9176943B2 (en) | 2008-05-12 | 2015-11-03 | Adobe Systems Incorporated | Comment presentation in electronic documents |
US9329744B2 (en) | 2008-05-12 | 2016-05-03 | Adobe Systems Incorporated | Segmented scroll bar |
US9418054B2 (en) | 2008-05-12 | 2016-08-16 | Adobe Systems Incorporated | Document comment management |
US10055392B2 (en) * | 2008-05-12 | 2018-08-21 | Adobe Systems Incorporated | History-based archive management |
US10628835B2 (en) | 2011-10-11 | 2020-04-21 | Consumeron, Llc | System and method for remote acquisition and deliver of goods |
US10629189B2 (en) | 2013-03-15 | 2020-04-21 | International Business Machines Corporation | Automatic note taking within a virtual meeting |
US10642478B2 (en) | 2017-04-10 | 2020-05-05 | Microsoft Technology Licensing Llc | Editable whiteboard timeline |
US10817169B2 (en) | 2016-10-14 | 2020-10-27 | Microsoft Technology Licensing, Llc | Time-correlated ink |
US11238465B2 (en) | 2009-08-26 | 2022-02-01 | Consumeron, Llc | System and method for remote acquisition and delivery of goods |
US20220308942A1 (en) * | 2018-07-06 | 2022-09-29 | Capital One Services, Llc | Systems and methods for censoring text inline |
US20220398224A1 (en) * | 2020-10-30 | 2022-12-15 | Docusign, Inc. | Edit Interface in an Online Document System |
Families Citing this family (126)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AUPQ582900A0 (en) * | 2000-02-24 | 2000-03-16 | Silverbrook Research Pty Ltd | Printed media production |
US7111009B1 (en) * | 1997-03-14 | 2006-09-19 | Microsoft Corporation | Interactive playlist generation using annotations |
US7596755B2 (en) | 1997-12-22 | 2009-09-29 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US7954056B2 (en) * | 1997-12-22 | 2011-05-31 | Ricoh Company, Ltd. | Television-based visualization and navigation interface |
US7689898B2 (en) | 1998-05-07 | 2010-03-30 | Astute Technology, Llc | Enhanced capture, management and distribution of live presentations |
US6917965B2 (en) * | 1998-09-15 | 2005-07-12 | Microsoft Corporation | Facilitating annotation creation and notification via electronic mail |
US6956593B1 (en) | 1998-09-15 | 2005-10-18 | Microsoft Corporation | User interface for creating, viewing and temporally positioning annotations for media content |
US7174083B1 (en) * | 1998-12-30 | 2007-02-06 | Xerox Corporation | Systems and methods for using structured representations to index recordings of activity |
US7330875B1 (en) * | 1999-06-15 | 2008-02-12 | Microsoft Corporation | System and method for recording a presentation for on-demand viewing over a computer network |
US6976032B1 (en) * | 1999-11-17 | 2005-12-13 | Ricoh Company, Ltd. | Networked peripheral for visitor greeting, identification, biographical lookup and tracking |
US7302490B1 (en) | 2000-05-03 | 2007-11-27 | Microsoft Corporation | Media file format to support switching between multiple timeline-altered media streams |
US20020049979A1 (en) * | 2000-05-18 | 2002-04-25 | Patrick White | Multiple camera video system which displays selected images |
US20020089587A1 (en) * | 2000-05-18 | 2002-07-11 | Imove Inc. | Intelligent buffering and reporting in a multiple camera data streaming video system |
AU2001264723A1 (en) * | 2000-05-18 | 2001-11-26 | Imove Inc. | Multiple camera video system which displays selected images |
US6714222B1 (en) * | 2000-06-21 | 2004-03-30 | E2 Home Ab | Graphical user interface for communications |
EP1323054A4 (en) * | 2000-08-29 | 2009-04-22 | Eloquent Inc | Tool for collaborative edit/search of dyanamic objects |
US6599130B2 (en) * | 2001-02-02 | 2003-07-29 | Illinois Institute Of Technology | Iterative video teaching aid with recordable commentary and indexing |
US20040027370A1 (en) * | 2001-02-15 | 2004-02-12 | Denny Jaeger | Graphic user interface and method for creating slide shows |
US6897880B2 (en) * | 2001-02-22 | 2005-05-24 | Sony Corporation | User interface for generating parameter values in media presentations based on selected presentation instances |
US7143353B2 (en) * | 2001-03-30 | 2006-11-28 | Koninklijke Philips Electronics, N.V. | Streaming video bookmarks |
US7454708B2 (en) * | 2001-05-25 | 2008-11-18 | Learning Tree International | System and method for electronic presentations with annotation of preview material |
US7131068B2 (en) * | 2001-05-25 | 2006-10-31 | Learning Tree International | System and method for electronic presentations having simultaneous display windows in a control screen |
US7747943B2 (en) * | 2001-09-07 | 2010-06-29 | Microsoft Corporation | Robust anchoring of annotations to content |
US7356763B2 (en) * | 2001-09-13 | 2008-04-08 | Hewlett-Packard Development Company, L.P. | Real-time slide presentation multimedia data object and system and method of recording and browsing a multimedia data object |
US20040205477A1 (en) * | 2001-09-13 | 2004-10-14 | I-Jong Lin | System for recording a presentation |
US20040237032A1 (en) * | 2001-09-27 | 2004-11-25 | David Miele | Method and system for annotating audio/video data files |
US7861169B2 (en) | 2001-11-19 | 2010-12-28 | Ricoh Co. Ltd. | Multimedia print driver dialog interfaces |
US7495795B2 (en) | 2002-02-21 | 2009-02-24 | Ricoh Company, Ltd. | Interface for printing multimedia information |
US7314994B2 (en) | 2001-11-19 | 2008-01-01 | Ricoh Company, Ltd. | Music processing printer |
US8635531B2 (en) | 2002-02-21 | 2014-01-21 | Ricoh Company, Ltd. | Techniques for displaying information stored in multiple multimedia documents |
US7415670B2 (en) * | 2001-11-19 | 2008-08-19 | Ricoh Co., Ltd. | Printer with audio/video localization |
US7747655B2 (en) | 2001-11-19 | 2010-06-29 | Ricoh Co. Ltd. | Printable representations for time-based media |
US7146574B2 (en) * | 2001-12-21 | 2006-12-05 | Microsoft Corporation | Systems and methods for interfacing with digital history data |
US7243301B2 (en) | 2002-04-10 | 2007-07-10 | Microsoft Corporation | Common annotation framework |
US20030202004A1 (en) * | 2002-04-30 | 2003-10-30 | I-Jong Lin | System and method for providing a low-bit rate distributed slide show presentation |
US20040008970A1 (en) * | 2002-07-09 | 2004-01-15 | Junkersfeld Phillip Aaron | Enhanced bookmarks for digital video playback |
US7257774B2 (en) * | 2002-07-30 | 2007-08-14 | Fuji Xerox Co., Ltd. | Systems and methods for filtering and/or viewing collaborative indexes of recorded media |
US7415667B2 (en) * | 2003-01-31 | 2008-08-19 | Ricoh Company, Ltd. | Generating augmented notes and synchronizing notes and document portions based on timing information |
US7251786B2 (en) * | 2003-02-26 | 2007-07-31 | Microsoft Corporation | Meeting information |
US7730407B2 (en) * | 2003-02-28 | 2010-06-01 | Fuji Xerox Co., Ltd. | Systems and methods for bookmarking live and recorded multimedia documents |
KR100547335B1 (en) * | 2003-03-13 | 2006-01-26 | 엘지전자 주식회사 | Video playing method and system, apparatus using the same |
JP2004280940A (en) * | 2003-03-14 | 2004-10-07 | Funai Electric Co Ltd | Disk player |
US20040205547A1 (en) * | 2003-04-12 | 2004-10-14 | Feldt Kenneth Charles | Annotation process for message enabled digital content |
US20040255232A1 (en) * | 2003-06-11 | 2004-12-16 | Northwestern University | Networked presentation system |
US7511846B2 (en) | 2003-09-25 | 2009-03-31 | Ricoh Co., Ltd. | Printer having embedded functionality for printing time-based media |
US7864352B2 (en) | 2003-09-25 | 2011-01-04 | Ricoh Co. Ltd. | Printer with multimedia server |
US8077341B2 (en) | 2003-09-25 | 2011-12-13 | Ricoh Co., Ltd. | Printer with audio or video receiver, recorder, and real-time content-based processing logic |
JP2005108230A (en) | 2003-09-25 | 2005-04-21 | Ricoh Co Ltd | Printing system with embedded audio/video content recognition and processing function |
US20050097451A1 (en) * | 2003-11-03 | 2005-05-05 | Cormack Christopher J. | Annotating media content with user-specified information |
JP4686990B2 (en) * | 2004-03-10 | 2011-05-25 | 富士ゼロックス株式会社 | Content processing system, content processing method, and computer program |
JP2005267279A (en) * | 2004-03-18 | 2005-09-29 | Fuji Xerox Co Ltd | Information processing system and information processing method, and computer program |
US8274666B2 (en) | 2004-03-30 | 2012-09-25 | Ricoh Co., Ltd. | Projector/printer for displaying or printing of documents |
US20050289453A1 (en) * | 2004-06-21 | 2005-12-29 | Tsakhi Segal | Apparatys and method for off-line synchronized capturing and reviewing notes and presentations |
JP2006148730A (en) * | 2004-11-24 | 2006-06-08 | Hitachi Omron Terminal Solutions Corp | Conference system and conference information providing method |
US20060161838A1 (en) * | 2005-01-14 | 2006-07-20 | Ronald Nydam | Review of signature based content |
US7551312B1 (en) * | 2005-03-17 | 2009-06-23 | Ricoh Co., Ltd. | Annotable document printer |
US7466858B2 (en) * | 2005-04-28 | 2008-12-16 | Fuji Xerox Co., Ltd. | Methods for slide image classification |
US20060244818A1 (en) * | 2005-04-28 | 2006-11-02 | Comotiv Systems, Inc. | Web-based conferencing system |
JP2006352879A (en) * | 2005-06-17 | 2006-12-28 | Fuji Xerox Co Ltd | Method of identifying and visualizing event in video frame, and system for generating timeline of event in video stream |
US7554576B2 (en) * | 2005-06-20 | 2009-06-30 | Ricoh Company, Ltd. | Information capture and recording system for controlling capture devices |
US7440123B2 (en) * | 2005-07-20 | 2008-10-21 | Eastman Kodak Company | Adaptive printing |
US9384178B2 (en) * | 2005-12-09 | 2016-07-05 | Adobe Systems Incorporated | Review of signature based content |
US8533199B2 (en) * | 2005-12-14 | 2013-09-10 | Unifi Scientific Advances, Inc | Intelligent bookmarks and information management system based on the same |
US7989720B2 (en) * | 2006-01-04 | 2011-08-02 | Fuji Xerox Co., Ltd. | Smart cable for controlling video capture device |
US20070198637A1 (en) * | 2006-01-04 | 2007-08-23 | Scott Deboy | Conferencing system with data file management |
US20070156829A1 (en) * | 2006-01-05 | 2007-07-05 | Scott Deboy | Messaging system with secure access |
US7911482B1 (en) | 2006-01-06 | 2011-03-22 | Videomining Corporation | Method and system for efficient annotation of object trajectories in image sequences |
US20070239827A1 (en) * | 2006-02-13 | 2007-10-11 | Scott Deboy | Global chat system |
WO2008026074A2 (en) * | 2006-03-14 | 2008-03-06 | Technische Universitat Darmstadt | Distributed interactive augmentation of display output |
US20070286366A1 (en) * | 2006-03-17 | 2007-12-13 | Scott Deboy | Chat presence system |
US7954049B2 (en) | 2006-05-15 | 2011-05-31 | Microsoft Corporation | Annotating multimedia files along a timeline |
US20070276910A1 (en) * | 2006-05-23 | 2007-11-29 | Scott Deboy | Conferencing system with desktop sharing |
US20070276852A1 (en) * | 2006-05-25 | 2007-11-29 | Microsoft Corporation | Downloading portions of media files |
US20070282793A1 (en) * | 2006-06-01 | 2007-12-06 | Majors Kenneth D | Computer desktop sharing |
US20080005245A1 (en) * | 2006-06-30 | 2008-01-03 | Scott Deboy | Conferencing system with firewall |
US20080043964A1 (en) * | 2006-07-14 | 2008-02-21 | Majors Kenneth D | Audio conferencing bridge |
US20080021968A1 (en) * | 2006-07-19 | 2008-01-24 | Majors Kenneth D | Low bandwidth chat system |
US8437409B2 (en) | 2006-12-06 | 2013-05-07 | Carnagie Mellon University | System and method for capturing, editing, searching, and delivering multi-media content |
US20080065727A1 (en) * | 2006-09-13 | 2008-03-13 | Majors Kenneth D | Conferencing system with improved access |
US20080066001A1 (en) * | 2006-09-13 | 2008-03-13 | Majors Kenneth D | Conferencing system with linked chat |
US20080065999A1 (en) * | 2006-09-13 | 2008-03-13 | Majors Kenneth D | Conferencing system with document access |
US20080104503A1 (en) * | 2006-10-27 | 2008-05-01 | Qlikkit, Inc. | System and Method for Creating and Transmitting Multimedia Compilation Data |
US7827490B2 (en) * | 2006-11-30 | 2010-11-02 | Microsoft Corporation | Media state user interface |
US20080222505A1 (en) * | 2007-01-08 | 2008-09-11 | David Chmura | Method of capturing a presentation and creating a multimedia file |
JP5092469B2 (en) * | 2007-03-15 | 2012-12-05 | ソニー株式会社 | Imaging apparatus, image processing apparatus, image display control method, and computer program |
US20080244442A1 (en) * | 2007-03-30 | 2008-10-02 | Microsoft Corporation | Techniques to share information between application programs |
US20080256114A1 (en) * | 2007-04-10 | 2008-10-16 | Microsoft Corporation | Techniques to display associated information between application programs |
US20080270761A1 (en) * | 2007-04-30 | 2008-10-30 | Microsoft Corporation | Techniques to generate event contexts for recurring events |
US8269728B2 (en) * | 2007-06-07 | 2012-09-18 | Smart Technologies Ulc | System and method for managing media data in a presentation system |
US8165451B2 (en) | 2007-11-20 | 2012-04-24 | Echostar Technologies L.L.C. | Methods and apparatus for displaying information regarding interstitials of a video stream |
US8165450B2 (en) | 2007-11-19 | 2012-04-24 | Echostar Technologies L.L.C. | Methods and apparatus for filtering content in a video stream using text data |
US8606085B2 (en) | 2008-03-20 | 2013-12-10 | Dish Network L.L.C. | Method and apparatus for replacement of audio data in recorded audio/video stream |
US20090249427A1 (en) * | 2008-03-25 | 2009-10-01 | Fuji Xerox Co., Ltd. | System, method and computer program product for interacting with unaltered media |
US8074173B2 (en) * | 2008-05-08 | 2011-12-06 | Microsoft Corporation | Associating input with computer based content |
US8156520B2 (en) | 2008-05-30 | 2012-04-10 | EchoStar Technologies, L.L.C. | Methods and apparatus for presenting substitute content in an audio/video stream using text data |
US20090317051A1 (en) * | 2008-06-18 | 2009-12-24 | Millington Daniel K | Mobile Timestamp Systems and Methods of Use |
US8639032B1 (en) * | 2008-08-29 | 2014-01-28 | Freedom Scientific, Inc. | Whiteboard archiving and presentation method |
KR101537592B1 (en) | 2008-09-03 | 2015-07-22 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
US8510771B2 (en) | 2008-12-24 | 2013-08-13 | Echostar Technologies L.L.C. | Methods and apparatus for filtering content from a presentation stream using signature data |
US8588579B2 (en) | 2008-12-24 | 2013-11-19 | Echostar Technologies L.L.C. | Methods and apparatus for filtering and inserting content into a presentation stream using signature data |
US8407735B2 (en) | 2008-12-24 | 2013-03-26 | Echostar Technologies L.L.C. | Methods and apparatus for identifying segments of content in a presentation stream using signature data |
US8255820B2 (en) | 2009-06-09 | 2012-08-28 | Skiff, Llc | Electronic paper display device event tracking |
US8437617B2 (en) | 2009-06-17 | 2013-05-07 | Echostar Technologies L.L.C. | Method and apparatus for modifying the presentation of content |
US8423088B2 (en) * | 2009-07-22 | 2013-04-16 | Microsoft Corporation | Aggregated, interactive communication timeline |
US8373741B2 (en) * | 2009-11-20 | 2013-02-12 | At&T Intellectual Property I, Lp | Apparatus and method for collaborative network in an enterprise setting |
US20110161834A1 (en) * | 2009-12-31 | 2011-06-30 | omNovia Technology, Inc. | Systems and Methods for Restoring and Recreating a Live On-Line Session |
US8934758B2 (en) * | 2010-02-09 | 2015-01-13 | Echostar Global B.V. | Methods and apparatus for presenting supplemental content in association with recorded content |
US20110202864A1 (en) * | 2010-02-15 | 2011-08-18 | Hirsch Michael B | Apparatus and methods of receiving and acting on user-entered information |
KR101781861B1 (en) * | 2011-04-04 | 2017-09-26 | 엘지전자 주식회사 | Image display device and method of displaying text in the same |
US9342516B2 (en) | 2011-05-18 | 2016-05-17 | Microsoft Technology Licensing, Llc | Media presentation playback annotation |
US10079039B2 (en) * | 2011-09-26 | 2018-09-18 | The University Of North Carolina At Charlotte | Multi-modal collaborative web-based video annotation system |
US9483109B2 (en) * | 2012-07-12 | 2016-11-01 | Spritz Technology, Inc. | Methods and systems for displaying text using RSVP |
US8955002B2 (en) * | 2013-01-16 | 2015-02-10 | Comcast Cable Communications, Llc | Tracking and responding to distracting events |
US20150206446A1 (en) * | 2014-01-22 | 2015-07-23 | Microsoft Technology Licensing, Llc. | Authoring, sharing, and consumption of online courses |
US9496922B2 (en) | 2014-04-21 | 2016-11-15 | Sony Corporation | Presentation of content on companion display device based on content presented on primary display device |
US9910840B2 (en) | 2015-04-03 | 2018-03-06 | Microsoft Technology Licensing, Llc | Annotating notes from passive recording with categories |
US10228775B2 (en) * | 2016-01-22 | 2019-03-12 | Microsoft Technology Licensing, Llc | Cross application digital ink repository |
US10950275B2 (en) * | 2016-11-18 | 2021-03-16 | Facebook, Inc. | Methods and systems for tracking media effects in a media effect index |
US10303928B2 (en) | 2016-11-29 | 2019-05-28 | Facebook, Inc. | Face detection for video calls |
US10554908B2 (en) | 2016-12-05 | 2020-02-04 | Facebook, Inc. | Media effect application |
US10579240B2 (en) * | 2018-02-09 | 2020-03-03 | Picmonkey, Llc | Live-rendered and forkable graphic edit trails |
US10063660B1 (en) | 2018-02-09 | 2018-08-28 | Picmonkey, Llc | Collaborative editing of media in a mixed computing environment |
US11102540B2 (en) * | 2019-04-04 | 2021-08-24 | Wangsu Science & Technology Co., Ltd. | Method, device and system for synchronously playing message stream and audio-video stream |
US11172269B2 (en) | 2020-03-04 | 2021-11-09 | Dish Network L.L.C. | Automated commercial content shifting in a video streaming system |
USD1002641S1 (en) * | 2020-05-15 | 2023-10-24 | Apple Inc. | Display screen or portion thereof with graphical user interface |
CN113038230B (en) * | 2021-03-10 | 2022-01-28 | 读书郎教育科技有限公司 | System and method for playing back videos and adding notes in intelligent classroom |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5717869A (en) * | 1995-11-03 | 1998-02-10 | Xerox Corporation | Computer controlled display system using a timeline to control playback of temporal data representing collaborative activities |
US5786814A (en) * | 1995-11-03 | 1998-07-28 | Xerox Corporation | Computer controlled display system activities using correlated graphical and timeline interfaces for controlling replay of temporal data representing collaborative activities |
US5828793A (en) * | 1996-05-06 | 1998-10-27 | Massachusetts Institute Of Technology | Method and apparatus for producing digital images having extended dynamic ranges |
US5852436A (en) * | 1994-06-30 | 1998-12-22 | Microsoft Corporation | Notes facility for receiving notes while the computer system is in a screen mode |
US5986655A (en) * | 1997-10-28 | 1999-11-16 | Xerox Corporation | Method and system for indexing and controlling the playback of multimedia documents |
US6128649A (en) * | 1997-06-02 | 2000-10-03 | Nortel Networks Limited | Dynamic selection of media streams for display |
US6173311B1 (en) * | 1997-02-13 | 2001-01-09 | Pointcast, Inc. | Apparatus, method and article of manufacture for servicing client requests on a network |
US6279014B1 (en) * | 1997-09-15 | 2001-08-21 | Xerox Corporation | Method and system for organizing documents based upon annotations in context |
US20030105812A1 (en) * | 2001-08-09 | 2003-06-05 | Gigamedia Access Corporation | Hybrid system architecture for secure peer-to-peer-communications |
US20030137536A1 (en) * | 2001-11-30 | 2003-07-24 | Hugh Harlan M. | Method and apparatus for communicating changes from and to a shared associative database using one-way communications techniques |
US6646655B1 (en) * | 1999-03-09 | 2003-11-11 | Webex Communications, Inc. | Extracting a time-sequence of slides from video |
US6732148B1 (en) * | 1999-12-28 | 2004-05-04 | International Business Machines Corporation | System and method for interconnecting secure rooms |
-
1999
- 1999-03-24 US US09/275,285 patent/US6452615B1/en not_active Expired - Lifetime
-
2000
- 2000-03-23 JP JP2000081135A patent/JP4171157B2/en not_active Expired - Fee Related
-
2002
- 2002-06-27 US US10/185,309 patent/US20020163548A1/en not_active Abandoned
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5852436A (en) * | 1994-06-30 | 1998-12-22 | Microsoft Corporation | Notes facility for receiving notes while the computer system is in a screen mode |
US5717869A (en) * | 1995-11-03 | 1998-02-10 | Xerox Corporation | Computer controlled display system using a timeline to control playback of temporal data representing collaborative activities |
US5786814A (en) * | 1995-11-03 | 1998-07-28 | Xerox Corporation | Computer controlled display system activities using correlated graphical and timeline interfaces for controlling replay of temporal data representing collaborative activities |
US5828793A (en) * | 1996-05-06 | 1998-10-27 | Massachusetts Institute Of Technology | Method and apparatus for producing digital images having extended dynamic ranges |
US6173311B1 (en) * | 1997-02-13 | 2001-01-09 | Pointcast, Inc. | Apparatus, method and article of manufacture for servicing client requests on a network |
US6128649A (en) * | 1997-06-02 | 2000-10-03 | Nortel Networks Limited | Dynamic selection of media streams for display |
US6279014B1 (en) * | 1997-09-15 | 2001-08-21 | Xerox Corporation | Method and system for organizing documents based upon annotations in context |
US5986655A (en) * | 1997-10-28 | 1999-11-16 | Xerox Corporation | Method and system for indexing and controlling the playback of multimedia documents |
US6646655B1 (en) * | 1999-03-09 | 2003-11-11 | Webex Communications, Inc. | Extracting a time-sequence of slides from video |
US6732148B1 (en) * | 1999-12-28 | 2004-05-04 | International Business Machines Corporation | System and method for interconnecting secure rooms |
US20030105812A1 (en) * | 2001-08-09 | 2003-06-05 | Gigamedia Access Corporation | Hybrid system architecture for secure peer-to-peer-communications |
US20030137536A1 (en) * | 2001-11-30 | 2003-07-24 | Hugh Harlan M. | Method and apparatus for communicating changes from and to a shared associative database using one-way communications techniques |
Cited By (70)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070038935A1 (en) * | 1999-11-17 | 2007-02-15 | Ricoh Company, Ltd. | Techniques for capturing information during multimedia presentations |
US7653925B2 (en) | 1999-11-17 | 2010-01-26 | Ricoh Company, Ltd. | Techniques for receiving information during multimedia presentations and communicating the information |
US7669127B2 (en) | 1999-11-17 | 2010-02-23 | Ricoh Company, Ltd. | Techniques for capturing information during multimedia presentations |
US7647555B1 (en) * | 2000-04-13 | 2010-01-12 | Fuji Xerox Co., Ltd. | System and method for video access from notes or summaries |
US7260771B2 (en) * | 2001-04-26 | 2007-08-21 | Fuji Xerox Co., Ltd. | Internet-based system for multimedia meeting minutes |
US20020161804A1 (en) * | 2001-04-26 | 2002-10-31 | Patrick Chiu | Internet-based system for multimedia meeting minutes |
US20030117532A1 (en) * | 2001-12-25 | 2003-06-26 | Seiko Epson Corporation | Projector wireless control system and wireless control method |
US6793352B2 (en) * | 2001-12-25 | 2004-09-21 | Seiko Epson Corporation | Projector wireless control system and wireless control method |
US6906741B2 (en) * | 2002-01-29 | 2005-06-14 | Palm, Inc. | System for and method of conferencing with a handheld computer using multiple media types |
US20030142200A1 (en) * | 2002-01-29 | 2003-07-31 | Palm, Inc. | System for and method of conferencing with a handheld computer using multiple media types |
US20070081075A1 (en) * | 2002-01-29 | 2007-04-12 | Palm, Inc. | Videoconferencing bandwidth management for a handheld computer system and method |
US20040139482A1 (en) * | 2002-10-25 | 2004-07-15 | Hale Greg B. | Streaming of digital data to a portable device |
US8634030B2 (en) * | 2002-10-25 | 2014-01-21 | Disney Enterprises, Inc. | Streaming of digital data to a portable device |
WO2004038982A3 (en) * | 2002-10-25 | 2004-07-08 | Greg B Hale | Streaming of digital data to a portable device |
WO2004038982A2 (en) * | 2002-10-25 | 2004-05-06 | Disney Enterprises, Inc. | Streaming of digital data to a portable device |
US20140143257A1 (en) * | 2002-12-11 | 2014-05-22 | Trio Systems Llc | Annotation system for creating and retrieving media and methods relating to same |
US9507776B2 (en) * | 2002-12-11 | 2016-11-29 | Trio Systems, Llc | Annotation system for creating and retrieving media and methods relating to same |
US8627193B2 (en) | 2003-01-09 | 2014-01-07 | Kaleidescape, Inc. | Bookmarks and watchpoints for selection and presentation of media streams |
US20120281967A1 (en) * | 2003-01-09 | 2012-11-08 | Kaleidescape, Inc. | Bookmarks and Watchpoints for Selection and Presentation of Media Streams |
US20050163212A1 (en) * | 2003-03-31 | 2005-07-28 | Safehouse International Limited | Displaying graphical output |
US20050125717A1 (en) * | 2003-10-29 | 2005-06-09 | Tsakhi Segal | System and method for off-line synchronized capturing and reviewing notes and presentations |
EP1536348A1 (en) * | 2003-11-26 | 2005-06-01 | Ricoh Company, Ltd. | Techniques for integrating note-taking and multimedia information |
US7689712B2 (en) | 2003-11-26 | 2010-03-30 | Ricoh Company, Ltd. | Techniques for integrating note-taking and multimedia information |
US20050122397A1 (en) * | 2003-12-03 | 2005-06-09 | Safehouse International Limited | Recording a sequence of images |
US20050163346A1 (en) * | 2003-12-03 | 2005-07-28 | Safehouse International Limited | Monitoring an output from a camera |
US7664292B2 (en) | 2003-12-03 | 2010-02-16 | Safehouse International, Inc. | Monitoring an output from a camera |
US20050163345A1 (en) * | 2003-12-03 | 2005-07-28 | Safehouse International Limited | Analysing image data |
GB2408880A (en) * | 2003-12-03 | 2005-06-08 | Safehouse Internat Inc | Observing monitored image data and highlighting incidents on a timeline |
US8953674B2 (en) | 2003-12-03 | 2015-02-10 | Lighthaus Logic Inc. | Recording a sequence of images using two recording procedures |
US8948245B2 (en) | 2003-12-03 | 2015-02-03 | Lighthaus Logic Inc. | Displaying graphical output representing the activity of a plurality of monitoring detection devices |
US7930637B2 (en) | 2004-01-08 | 2011-04-19 | International Business Machines Corporation | Intelligent agenda object for a presentation application |
US20050154995A1 (en) * | 2004-01-08 | 2005-07-14 | International Business Machines Corporation | Intelligent agenda object for showing contextual location within a presentation application |
US20090300501A1 (en) * | 2004-01-08 | 2009-12-03 | International Business Machines Corporation | Intelligent agenda object for a presentation application |
US7620896B2 (en) | 2004-01-08 | 2009-11-17 | International Business Machines Corporation | Intelligent agenda object for showing contextual location within a presentation application |
US8805929B2 (en) | 2005-06-20 | 2014-08-12 | Ricoh Company, Ltd. | Event-driven annotation techniques |
US8464164B2 (en) * | 2006-01-24 | 2013-06-11 | Simulat, Inc. | System and method to create a collaborative web-based multimedia contextual dialogue |
US20120194523A1 (en) * | 2007-05-29 | 2012-08-02 | Livescribe, Inc. | Animation of Audio Ink |
US8194081B2 (en) * | 2007-05-29 | 2012-06-05 | Livescribe, Inc. | Animation of audio ink |
US20090027400A1 (en) * | 2007-05-29 | 2009-01-29 | Jim Marggraff | Animation of Audio Ink |
US8358309B2 (en) * | 2007-05-29 | 2013-01-22 | Livescribe, Inc. | Animation of audio ink |
US10055392B2 (en) * | 2008-05-12 | 2018-08-21 | Adobe Systems Incorporated | History-based archive management |
US8290902B1 (en) | 2008-05-12 | 2012-10-16 | Adobe Systems Incorporated | Shared edit access of electronic content |
US9176943B2 (en) | 2008-05-12 | 2015-11-03 | Adobe Systems Incorporated | Comment presentation in electronic documents |
US9418054B2 (en) | 2008-05-12 | 2016-08-16 | Adobe Systems Incorporated | Document comment management |
US9329744B2 (en) | 2008-05-12 | 2016-05-03 | Adobe Systems Incorporated | Segmented scroll bar |
US8768885B2 (en) | 2008-05-12 | 2014-07-01 | Adobe Systems Incorporated | Shared edit access of electronic content |
US9747017B2 (en) | 2008-05-12 | 2017-08-29 | Adobe Systems Incorporated | Segmented scroll bar |
US7949633B1 (en) | 2008-05-12 | 2011-05-24 | Adobe Systems Incorporated | Shared edit access of electronic content |
US8849869B2 (en) | 2008-05-12 | 2014-09-30 | Adobe Systems Incorporated | System and method for editing an item list in electronic content |
US8396900B1 (en) | 2008-05-12 | 2013-03-12 | Adobe Systems Incorporated | System and method for editing an item list in electronic content |
US7945595B1 (en) | 2008-05-12 | 2011-05-17 | Adobe Systems Incorporated | System and method for generating an item list in electronic content |
US8996621B2 (en) | 2008-05-12 | 2015-03-31 | Adobe Systems Incorporated | Asynchronous comment updates |
US8380866B2 (en) | 2009-03-20 | 2013-02-19 | Ricoh Company, Ltd. | Techniques for facilitating annotations |
US11238465B2 (en) | 2009-08-26 | 2022-02-01 | Consumeron, Llc | System and method for remote acquisition and delivery of goods |
US20110119588A1 (en) * | 2009-11-17 | 2011-05-19 | Siracusano Jr Louis H | Video storage and retrieval system and method |
US8806341B2 (en) * | 2009-12-10 | 2014-08-12 | Hulu, LLC | Method and apparatus for navigating a media program via a histogram of popular segments |
US20110145726A1 (en) * | 2009-12-10 | 2011-06-16 | Hulu Llc | Method and apparatus for navigating a media program via a histogram of popular segments |
US8706898B2 (en) | 2009-12-10 | 2014-04-22 | Hulu, LLC | Navigating a video, a transcript of a dialog, and a histogram |
US10628835B2 (en) | 2011-10-11 | 2020-04-21 | Consumeron, Llc | System and method for remote acquisition and deliver of goods |
US9529884B2 (en) | 2012-01-19 | 2016-12-27 | Microsoft Technology Licensing, Llc | Usage based synchronization of note-taking application features |
WO2013109510A1 (en) * | 2012-01-19 | 2013-07-25 | Microsoft Corporation | Usage based synchronization of note-taking application features |
US10629189B2 (en) | 2013-03-15 | 2020-04-21 | International Business Machines Corporation | Automatic note taking within a virtual meeting |
US10629188B2 (en) | 2013-03-15 | 2020-04-21 | International Business Machines Corporation | Automatic note taking within a virtual meeting |
JP2015079494A (en) * | 2013-10-15 | 2015-04-23 | シャープ株式会社 | Information processor, information processing method, and recording medium |
US9323447B2 (en) | 2013-10-15 | 2016-04-26 | Sharp Laboratories Of America, Inc. | Electronic whiteboard and touch screen method for configuring and applying metadata tags thereon |
US20150116283A1 (en) * | 2013-10-24 | 2015-04-30 | Livescribe Inc. | Paper Strip Presentation Of Grouped Content |
US10817169B2 (en) | 2016-10-14 | 2020-10-27 | Microsoft Technology Licensing, Llc | Time-correlated ink |
US10642478B2 (en) | 2017-04-10 | 2020-05-05 | Microsoft Technology Licensing Llc | Editable whiteboard timeline |
US20220308942A1 (en) * | 2018-07-06 | 2022-09-29 | Capital One Services, Llc | Systems and methods for censoring text inline |
US20220398224A1 (en) * | 2020-10-30 | 2022-12-15 | Docusign, Inc. | Edit Interface in an Online Document System |
Also Published As
Publication number | Publication date |
---|---|
JP2000341660A (en) | 2000-12-08 |
JP4171157B2 (en) | 2008-10-22 |
US6452615B1 (en) | 2002-09-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6452615B1 (en) | System and apparatus for notetaking with digital video and ink | |
US7730407B2 (en) | Systems and methods for bookmarking live and recorded multimedia documents | |
US7647555B1 (en) | System and method for video access from notes or summaries | |
Chiu et al. | NoteLook: Taking notes in meetings with digital video and ink | |
Mackay | EVA: An experimental video annotator for symbolic analysis of video data | |
US9837077B2 (en) | Enhanced capture, management and distribution of live presentations | |
Cruz et al. | Capturing and playing multimedia events with STREAMS | |
US7266568B1 (en) | Techniques for storing multimedia information with source documents | |
Chiu et al. | LiteMinutes: an Internet-based system for multimedia meeting minutes | |
US7598975B2 (en) | Automatic face extraction for use in recorded meetings timelines | |
JP5140949B2 (en) | Method, system and apparatus for processing digital information | |
JP3185505B2 (en) | Meeting record creation support device | |
US6014183A (en) | Method and apparatus for detecting scene changes in a digital video stream | |
Chiu et al. | Room with a rear view. Meeting capture in a multimedia conference room | |
US20110072037A1 (en) | Intelligent media capture, organization, search and workflow | |
US20070195159A1 (en) | Method and system for audio/video capturing, streaming, recording and playback | |
Erol et al. | An overview of technologies for e-meeting and e-lecture | |
US20020188772A1 (en) | Media production methods and systems | |
Gleicher et al. | A framework for virtual videography | |
JP4686990B2 (en) | Content processing system, content processing method, and computer program | |
Zhang et al. | A presentation authoring tool for media devices distributed environments | |
JP2005260511A (en) | System and method for processing content, and computer program | |
Ichimura | Delivering chalk talks on the internet | |
JP2005260512A (en) | System and method for processing content, and computer program | |
Li et al. | Frame-buffer on demand: Applications of stateless client systems in web-based learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: JPMORGAN CHASE BANK, AS COLLATERAL AGENT, TEXAS Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:015134/0476 Effective date: 20030625 Owner name: JPMORGAN CHASE BANK, AS COLLATERAL AGENT,TEXAS Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:015134/0476 Effective date: 20030625 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |