arXiv:1307.3040v2  [cs.AI]  31 Mar 2014
Between Sense and Sensibility
Declarative narrativisation of mental models as a basis and benchmark for visuo-spatial cognition and
computation focussed collaborative cognitive systems
Mehul Bhatt
Cognitive Systems, and
Spatial Cognition Research Center
University of Bremen, Germany
bhatt@informatik.uni-bremen.de
www.mehulbhatt.org
subject keywords
computer science; cognitive science; artiﬁcial intelligence; cognitive systems; human-computer
interaction
general keywords
computational models of narrative; spatial cognition and computation; commonsense reasoning; spatial
and temporal reasoning; action and change; cognitive robotics; design, architecture, urban planning;
geographic information systems; spatial assistance systems
2
M. Bhatt / Between Sensing and Sensibility
POSITION STATEMENT
What lies between ‘sensing’ and ‘sensibility’? In other words, what kind of cognitive processes mediate
sensing capability, and the formation of sensible impressions —e.g., abstractions, analogies, hypotheses
and theory formation, beliefs and their revision, argument formation— in domain-speciﬁc problem solv-
ing, or in regular activities of everyday living, working and simply going around in the environment?
How can knowledge and reasoning about such capabilities, as exhibited by humans in particular problem
contexts, be used as a model and benchmark for the development of collaborative cognitive (interaction)
systems concerned with human assistance, assurance, and empowerment?
We pose these questions in the context of a range of assistive technologies concerned with visuo-spatial
perception and cognition tasks encompassing aspects such as commonsense, creativity, and the application
of specialist domain knowledge and problem-solving thought processes [Bhatt et al., 2013a]. Assistive
technologies being considered include: (a) human activity interpretation from sensor data; (b) high-level
cognitive control for human-robot collaboration; (c) people-centred, function-driven, creative design in
domains such as architecture & digital media creation, and (d) high-level qualitative analyses in spatio-
temporal data-intensive geographic information systems
A.
Compuational Narrativisation as a Benchmark
Narrativisation processes pertaining to space, actions, and change are ubiquitous [Bhatt, 2012, Tversky,
2004]. Humans, robots, and systems involving action and mutual (computer-human) interaction are em-
bedded in space. Space, spatial conﬁgurations, and in effect, their perceptually grounded mental models
[Johnson-Laird, 1983], declarative abstractions, or ad-hoc system-level representations undergo change
— the notion of time, or change-based temporal progression, and therefore spatio-temporal dynamics and
spatio-temporal narrativisation (processes) arise naturally.
The signiﬁcance of narratives in everyday discourse, interpretation, interaction, belief formation, and
decision-making has been acknowledged and studied in a range of scientiﬁc, humanistic, and artistic dis-
ciplines. Narrativisation of everyday perceptions by humans, and the signiﬁcance of narratives, e.g., in
communication and interaction, has been investigated under several frameworks, and through several in-
terdisciplinary initiatives involving the arts, humanities, and social sciences, e.g., the narrative paradigm
[Fisher, 1987], narrative analysis [Riessman, 1993], narratology [Herman et al., 2005, Meister, 2011,
Prince, 1982], discourse analysis and computational narratology [Barthes and Duisit, 1975, Goguen, 2004,
Lakoff and Narayanan, 2010, Mani, 2012, 2013]. Broadly, the study of narratives has attracted attention
from several quarters, most prominently in disciplines such as literature, linguistics, anthropology, semi-
otics, cultural studies, geography, psychology, cognitive science, logic, and computer science.
We regard narratives, and high-level processes of narrativisation emanating therefrom, as a general under-
lying structure serving the crucial function of perceptual sense-making — i.e., as a link between problem-
speciﬁc perceptual sensing and the (computational) formation of sensible impressions concerned with in-
terpretation and analytical tasks. Given the nature of the visuo-spatial cognition tasks being considered,
the particular form of the proposed narrative structure is that of cognitively inspired computational model
of narrativisation involving high-level commonsense reasoning with space, events, actions, change, and
interaction. In particular, the following capabilities are crucial:
M. Bhatt / Between Sensing and Sensibility
3
1. Narrativising—based on modalities such as language, diagrams, visual abstractions, analogy, concep-
tual blending, and hypotheses— certain aspects of perceived reality, e.g., obtained via 2D or 3D per-
ception sensors, from large-scale complex datasets representing spatio-temporal data etc
2. Achieving the narrativisation —or human-like perceptual sense-making— with a level of declarative
abstraction,1 analytical accuracy, and semantic & descriptive quality and expressibility that is compa-
rable with, or even outperforming, expected human performance in speciﬁc problem contexts.
Computational narratives not only provide a rich cognitive basis, but they also serve as a benchmark of
functional performance in our development of computational cognitive assistance systems. We posit that
computational narrativisation pertaining to space, actions, and change provides a useful model of visual
and spatio-temporal thinking within a wide-range of problem-solving tasks and application areas (see B;
B1-B5) where collaborative cognitive systems could serve an assistive and empowering function.
B.
Narrativised Spatio-Temporal Thinking in Analytical and Creative Problem Solving
The ubiquity and diversity of visuo-spatial narrativisation is self-evident in a range of cognitive (interac-
tion) systems and technologies that aim to assist and empower people in creative, specialist, and regular
everyday living situations. Consider assistive technologies and systems concerning the problem domain
and application areas in (B1–B5):
B1.
Perceptual Narratives and Human Activity Interpretation
Perceptual narratives pertain to artiﬁcial sensor grounded visual, auditory, and haptic observations in the
real world. Declarative models of perceptual narratives can be used for assistive tasks in the course of
activities in everyday life and work (e.g., human activity interpretation, semantic model generation from
video, ambient intelligence) [Bhatt et al., 2013b,c].
B2.
Narrative-based Control for Cognitive Robotics
To the extent that a perceptual narrative, as in (A1), can help to make sense of a situation for an artiﬁcial
agent (e.g., robot, system), it can also be the basis of future action. For instance, a robotic agent can identify
abnormalities in a narrative, which may in turn affect subsequent (re)planning and sensing behaviour, and
dialog with a user [Eppe and Bhatt, 2013, Suchan and Bhatt, 2014]
B3.
Narratives of User Experience for Spatial Design & Architecture
Narratives of user experience pertain to human-grounded visuo-spatial and locomotive modalities. Con-
sider this: you enter a Museum for the ﬁrst time; as you go around, guided by the internal structure of the
building, you form a narrative of user experience based on your visuo-spatial, locomotive, and affordance-
based perceptions in the Museum. Architects concerned with designing this Museum are confronted with
anticipating this sort of narrative of user experience, at a time when all that exists during the initial design
conception phase is ‘empty space’ [Bhatt et al., 2012, 2014].
B4.
Narratives for Creative Assistance in Digital and New Media Production
We interpret creative assistance in digital media production as the capability of computational tools to sup-
port the creative skills of experts and artists at several stages within the media design and creation process.
Consider the domain of ﬁlm, animation, and comic book pre-production. Here, one may identify several
forms of assistance at the production phase, e.g., virtual cinematography, storyboarding, and scene visual-
isation from scripts and automatic camera control in the animation domain. Professionals who may be as-
sisted, in a ﬁlm context, include: cinematographers, directors, script and screenplay writers, storyboarding
artists [Bhatt and Flanagan, 2010].
1Declarative-ness signiﬁes the existence of models that can be reasoned and queried upon, e.g., within a traditional declarative
knowledge representation and reasoning framework such as logic programming, constraint logic programming, description logic
based conceptual reasoning, answer-set programming, and specialised commonsense reasoners based on expressive action logics.
Details of such underlying enabling methods are not reelvant for this position statement.
4
M. Bhatt / Between Sensing and Sensibility
B5.
Geospatial Narratives and their Spatio-Temporal Dynamics
Geospatial narratives attempt to make sense of massive quantities of micro and macro-level spatio-
temporal data pertaining to environmental, socio-economic and demographic processes operating in
a geospatial context [Bhatt and Wallgruen, 2011, 2014]. Such narratives pertain to spatio-temporal
databases of precise measurements about environmental features, aerial imagery, sensor network databases
with real-time information about natural and artiﬁcial processes and phenomena etc. Geospatial narratives
typically span a temporal horizon encompassing generational change, but these could also pertain to the
scale of everyday ‘life in the city’, natural environmental processes etc. [Bhatt and Wallgruen, 2014]
In the backdrop of the problem domains in (B1–B5), we are investigating:
(a) the conceptual and com-
putational aspects of narrative-based visuo-spatial cognition, (b) declarative model of narrative knowl-
edge, and its relationship with spatio-linguistically grounded behavioural and formal theories, (c) role of
specialised knowledge representation and reasoning mechanisms as underlying methods for automating
high-level narrativisation processes from the viewpoint of visuo-spatial cognition and creativity.
Acknowledgements
I gratefully acknowledge the funding and support of the German Research Foundation (DFG),
www.dfg.de/ — part of the work described in this paper has been conducted as part of the DFG funded
SFB/TR 8 Spatial Cognition project [DesignSpace], http://www.design-space.org.
I thank Manfred Eppe, Carl Schultz, Jakob Suchan, and Jan Oliver Wallgr¨un for constructive discussions
and collaborations related to different aspects of the challenges being addressed / positioned in this ab-
stract.
Related Work
R. Barthes and L. Duisit. An Introduction to the Structural Analysis of Narrative. New Literary History, 6(2):237–272, 1975.
ISSN 00286087. doi: 10.2307/468419. URL http://dx.doi.org/10.2307/468419.
M. Bhatt. Reasoning about Space, Actions and Change: A Paradigm for Applications of Spatial Reasoning. In Qualitative Spatial
Representation and Reasoning: Trends and Future Directions. IGI Global, USA, 2012. ISBN ISBN13: 9781616928681.
M. Bhatt and G. Flanagan. Spatio-temporal abduction for scenario and narrative completion. In M. Bhatt, H. Guesgen, and
S. Hazarika, editors, Proceedings of the International Workshop on Spatio-Temporal Dynamics, co-located with ECAI-10,
pages 31–36. ECAI Workshop Proceedings., August 2010.
M. Bhatt and J. O. Wallgruen. Analytical intelligence for geospatial dynamics. In Proceedings of COSIT 2011: Conference on
Spatial Information Theory, 2011.
M. Bhatt and J. O. Wallgruen.
Geospatial narratives and their spatio-temporal dynamics: Commonsense reasoning
for high-level analyses in geographic information systems.
Special Issue on: Geospatial Monitoring and Mod-
elling of Environmental Change, ISPRS International Journal of Geo-Information, 2014.
ISSN 2220-9964.
URL
http://arxiv.org/abs/1307.2541. In Press.
M. Bhatt, C. Schultz, and M. Huang. The Shape of Empty Space: Human-centred Cognitive Foundations in Computing for
Spatial Design. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 33–40, 2012.
M. Bhatt, C. Schultz, and C. Freksa. The ‘Space’ in Spatial Assistance Systems: Conception, Formalisation and Computation.
In T. Tenbrink, J. Wiener, and C. Claramunt, editors, Representing space in cognition: Interrelations of behavior, language,
and formal models. Series: Explorations in Language and Space. 978-0-19-967991-1, Oxford University Press, 2013a.
M. Bhatt, J. Suchan, and C. Freksa. ROTUNDE – A Smart Meeting Cinematography Initiative. In M. Bhatt, H. Guesgen, and
D. Cook, editors, Proceedings of the AAAI-2013 Workshop on Space, Time, and Ambient Intelligence (STAMI)., Washington,
US, 2013b. AAAI Press.
M. Bhatt / Between Sensing and Sensibility
5
M. Bhatt, J. Suchan, and C. Schultz. Cognitive Interpretation of Everyday Activities – Toward Perceptual Narrative Based Visuo-
Spatial Scene Interpretation. In M. Finlayson, B. Fisseni, B. Loewe, and J. C. Meister, editors, Computational Models of
Narrative (CMN) 2013., a satellite workshop of CogSci 2013: The 35th meeting of the Cognitive Science Society., Dagstuhl,
Germany, 2013c. OpenAccess Series in Informatics (OASIcs).
M. Bhatt, C. Schultz, and M. Thosar.
Computing narratives of cognitive user experience for building design analysis: Kr
for industry scale computer-aided architecture design. In T. Eiter, C. Baral, and G. D. Giacomo, editors, Principles of
Knowledge Representation and Reasoning: Proceedings of the 14th International Conference, KR 2014, 2014.
URL
http://kr.org/KR2014/. to appear.
M. Eppe and M. Bhatt. Narrative based Postdictive Reasoning for Cognitive Robotics. In COMMONSENSE 2013: 11th Interna-
tional Symposium on Logical Formalizations of Commonsense Reasoning, 2013.
W. R. Fisher. Human communication as narration: Toward a philosophy of reason, value, and action. University of South
Carolina Press, Columbia, SC, 1987.
J. Goguen. CSE 87C Winter 2004 Freshman Seminar on Computational Narratology. . New Literary History, 2004. URL
http://cseweb.ucsd.edu/˜goguen/courses/87w04/1.html.
D. Herman, M. Jahn, and M.-L. Ryan. Routledge Encyclopedia Of Narrative Theory. Routledge, Feb. 2005. ISBN 0415282594.
P. N. Johnson-Laird. Mental models: towards a cognitive science of language, inference, and consciousness. Harvard University
Press, Cambridge, MA, USA, 1983. ISBN 0-674-56882-6.
G.
Lakoff
and
S.
Narayanan.
Toward
a
computational
model
of
narrative,
2010.
URL
http://www.aaai.org/ocs/index.php/FSS/FSS10/paper/view/2323.
I. Mani. Computational modeling of narrative. Synthesis Lectures on Human Language Technologies, 5(3):1–142, 2012.
I. Mani.
Computational Narratology.
In P. H¨uhn, J. C. Meister, J. Pier, and W. Schmid, editors, The Living Handbook of
Narratology. Hamburg University Press, 2013.
J. C. Meister. Narratology. In P. H¨uhn, J. C. Meister, J. Pier, and W. Schmid, editors, The Living Handbook of Narratology.
Hamburg University Press, 2011.
G. Prince. Narratology: The Form and Function of Narrative. Mouton, 1982.
C. K. Riessman. Narrative Analysis. Newbury Park: Sage Publications, 1993.
J. Suchan and M. Bhatt. The ExpCog Framework: High-Level Spatial Control and Planning for Cognitive Robotics. In Bridges
between the Methodological and Practical Work of the Robotics and Cognitive Systems Communities - From Sensors to
Concepts. Intelligent Systems Reference Library, Springer, 2014. (in press).
B. Tversky. Narratives of Space, Time, and Life. Mind & Language, 19(4):380–392, 2004. doi: 10.1111/j.0268-1064.2004.
00264.x. URL http://dx.doi.org/10.1111/j.0268-1064.2004.00264.x.