{ "cells": [ { "cell_type": "markdown", "id": "39ece775", "metadata": {}, "source": [ "# Overview & Quick Start" ] }, { "cell_type": "markdown", "id": "cd7a22d7", "metadata": {}, "source": [ "This tutorial provides an introduction to usage of the CIMantic Graphs library (aka CIM-Graph). \n", "\n", "CIMantic Graphs is an open-source library for creating in-memory labeled property graphs for creating, parsing, and editing CIM power system models. It creates Python object instances in memory using a data profile exported from a specified CIM profile (e.g. IEC61970cim17v40 or GridAPPS-D RC4_2021)." ] }, { "cell_type": "markdown", "id": "ac891701", "metadata": {}, "source": [ "To install CIMantic Graphs clone the github repository or use pip install: `pip install cim-graph`" ] }, { "cell_type": "markdown", "id": "b0280e92", "metadata": {}, "source": [ "## Table of Contents:\n", "\n", "* [1. CIMantic Graphs Structure](#cimantic-graphs-structure)\n", "\n", "* [2. Importing CIMantic Graphs](#importing-cimantic-graphs)\n", "\n", "* [3. Specifying Connection Parameters](#specifying-connection-parameters)\n", "\n", "* [4. Connecting to the Data Source](#connecting-to-the-data-source)\n", "\n", "* [5. Creating a Container and Graph Model](#creating-a-container-and-graph-model)\n", "\n", "* [6. Automated Database Query Generation](#automated-database-queries)\n", "\n", "* [7. Traversing the Knowledge Graph](#traversing-the-knowledge-graph)\n", "\n", "* [8. Utils Shortcut Methods](#utils-shortcuts)" ] }, { "cell_type": "markdown", "id": "4860c0b8", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "17deb4d2", "metadata": {}, "source": [ "## CIMantic Graphs Structure\n", "\n", "CIMantic Graphs uses the layered architecture shown below:" ] }, { "cell_type": "markdown", "id": "853342b6", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "6bca6c88", "metadata": {}, "source": [ "### Database Layer\n", "CIMantic Graphs currently supports the following databases and interfaces:\n", "\n", "* Blazegraph Database\n", "\n", "* GraphDB Database\n", "\n", "* Neo4J Database\n", "\n", "* MySQL Database (in progress)\n", "\n", "* GridAPPS-D Platform\n", "\n", "* AVEVA PI Asset Framework (in progress)\n", "\n", "* RDFLib File Parser\n", "\n", "* XML Flat Files\n", "\n", "* JSON-LD Flat Files (in progress)\n", "\n", "* CSV Flat Files (in progress)\n", "\n", "The library uses a unified syntax for all upper-level calls and routines. The databases can be swapped interchangeably by changing the environment variables specified during application startup, with no other changes to any other application syntax or methods" ] }, { "cell_type": "markdown", "id": "2f713cc3", "metadata": {}, "source": [ "### Data Profile Layer\n", "\n", "CIMantic Graphs is able to support any standard or custom CIM profile. The CIM profile needs to be exported as an XSD data profile / schema. CIMantic Graphs is then able to ingest the data profile and convert all UML classes and attributes to python dataclasses, which power all of the routines and unified API syntax" ] }, { "cell_type": "markdown", "id": "48561db2", "metadata": {}, "source": [ "### API Layer\n", "\n", "CIMantic Graphs offers a breakthrough in terms of ease-of-use through a unified API with two core methods.\n", "\n", "__Access to labeled property graph objects:__\n", "\n", "* `network.graph[cim.ClassName]`: This offers access to a catalog of CIM object instances stored in memory and sorted by class type and mRID forming the named property graph.\n", "\n", "__Universal database query method:__\n", "\n", "* `network.get_all_edges(cim.ClassName)`: This is a universal query method that gets all attributes and all objects one edge away from instances of the specified class. This method works for all CIM classes, CIM profiles, serialization formats, and supported databases." ] }, { "cell_type": "markdown", "id": "f2baa96a", "metadata": {}, "source": [ "### Knowledge Graph Layer\n", "\n", "CIMantic Graphs offers three core knowledge graph classes for handing various kinds of power system models:\n", "\n", "* `BusBranchModel`: Transmission bus-branch models commonly used for planning and power flow studies\n", "\n", "* `NodeBreakerModel`: Transmission node-breaker models commonly used inside energy management systems\n", "\n", "* `FeederModel`: Distribution feeder models with support for single-phase unbalanced networks used in North America.\n", "\n", "Centralized or distributed representations of the power system network model can be used. Centralized models use a single labeled property graph for the network. Distributed models use nested `DistributedArea` class instances to represent the grouping of equipment inside a Substation, VoltageLevel, Bay, Feeder, and switch-delimited topological area inside a combined T+D model." ] }, { "cell_type": "markdown", "id": "7e50e57f", "metadata": {}, "source": [ "### Application Layer\n", "\n", "T+D applications are able to access all of the power system objects through knowledge graph, without any need to connect to the database or perform any custom i/o operations. " ] }, { "cell_type": "markdown", "id": "c82d5ab2", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "225b514c", "metadata": {}, "source": [ "## Importing CIMantic Graphs\n", "\n", "CIMantic Graphs contains several modules within the core library:\n", "\n", "* `data_profiles`: This package contains full CIM profiles exported through [CIMTool](https://github.com/CIMug-org/CIMTool/releases) and provides direct access to CIM dataclasses, their attributes, and usage documentation.\n", "\n", "* `databases`: This package contains database i/o connections and backend query handling for multiple databases (e.g. Blazegraph, GraphDB, Neo4J, MySQL, etc.).\n", "\n", "* `models`: This package contains knowledge graph models for transmission node-breaker, transmission bus-branch, and distribution feeder models.\n", "\n", "* `queries`: This package contains generic queries that are built at runtime for any CIM profile and CIM class.\n", "\n", "* `utils`: This package contains shortcut methods for common routines, such as writing out new files." ] }, { "cell_type": "markdown", "id": "ece42a4f", "metadata": {}, "source": [ "### Importing the CIM Profile\n", "The first step in using CIMantic Graphs is to import the python data profile for desired CIM profile. The data profile provides the ability to invoke CIM classes directly based on their name.\n", "\n" ] }, { "cell_type": "markdown", "id": "a11bf413", "metadata": {}, "source": [ "__Method 1:__ Directly import the desired CIM profile:" ] }, { "cell_type": "code", "execution_count": 1, "id": "f72f6871", "metadata": {}, "outputs": [], "source": [ "import cimgraph.data_profile.cim17v40 as cim" ] }, { "cell_type": "markdown", "id": "84c8e710", "metadata": {}, "source": [ "__Method 2:__ Use `importlib`:" ] }, { "cell_type": "code", "execution_count": 2, "id": "ac2d5fee", "metadata": {}, "outputs": [], "source": [ "import importlib\n", "cim_profile = 'cimhub_2023'\n", "cim = importlib.import_module('cimgraph.data_profile.' + cim_profile)" ] }, { "cell_type": "markdown", "id": "14beafa1", "metadata": {}, "source": [ "### Invoking CIM classes" ] }, { "cell_type": "markdown", "id": "748182ce", "metadata": {}, "source": [ "After importing the data profile, it is possible to create an instance of a class or view the attributes of any class. " ] }, { "cell_type": "markdown", "id": "9ef22b58", "metadata": {}, "source": [ "__Example 1:__ Create a new breaker with a name. If an mRID string is not provided, a UUID is automatically created based on the name or a random seed. Attributes of the class can be assigned " ] }, { "cell_type": "code", "execution_count": 3, "id": "c80ca66c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\",\n", " \"name\": \"breaker_01\",\n", " \"open\": \"True\",\n", " \"NormalOpen\": false\n", "}\n" ] } ], "source": [ "breaker = cim.Breaker(name = \"breaker_01\", open=True)\n", "breaker.NormalOpen = False\n", "breaker.pprint()" ] }, { "cell_type": "markdown", "id": "f196508e", "metadata": {}, "source": [ "__Example 2:__ Associate two CIM objects based on their associations" ] }, { "cell_type": "code", "execution_count": 4, "id": "d9cc48f3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\",\n", " \"name\": \"breaker_01\",\n", " \"EquipmentContainer\": {\n", " \"@id\": \"5d58b92a-5734-4ae7-8f3c-1357e662cf16\",\n", " \"@type\": \"Substation\"\n", " },\n", " \"open\": \"True\",\n", " \"NormalOpen\": false\n", "}\n", "{\n", " \"@id\": \"5d58b92a-5734-4ae7-8f3c-1357e662cf16\",\n", " \"@type\": \"Substation\",\n", " \"name\": \"sub_1\",\n", " \"Equipments\": [\n", " {\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\"\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "substation = cim.Substation(name = \"sub_1\")\n", "breaker.EquipmentContainer = substation\n", "substation.Equipments.append(breaker)\n", "breaker.pprint()\n", "substation.pprint()" ] }, { "cell_type": "markdown", "id": "28f96a79", "metadata": {}, "source": [ "__Example 3:__ View documentation of the ACLineSegment class" ] }, { "cell_type": "code", "execution_count": 5, "id": "a01ec261", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " A wire or combination of wires, with consistent electrical characteristics,\n", " building a single electrical system, used to carry alternating current\n", " between points in the power system.\n", " For symmetrical, transposed 3ph lines, it is sufficient to use attributes\n", " of the line segment, which describe impedances and admittances for the\n", " entire length of the segment. Additionally impedances can be computed by\n", " using length and associated per length impedances.\n", " The BaseVoltage at the two ends of ACLineSegments in a Line shall have\n", " the same BaseVoltage.nominalVoltage. However, boundary lines may have slightly\n", " different BaseVoltage.nominalVoltages and variation is allowed. Larger\n", " voltage difference in general requires use of an equivalent branch.\n", " \n" ] } ], "source": [ "print(cim.ACLineSegment.__doc__)" ] }, { "cell_type": "markdown", "id": "7f4eec9b", "metadata": {}, "source": [ "-----" ] }, { "cell_type": "markdown", "id": "4dbc75d2", "metadata": {}, "source": [ "## Specifying Environment Variables\n", "\n", "The next step in using any of the CIMantic Graphs library classes to establish the connection parameters for reading or writing the CIM model through a set of environment variables. The specific set of environment variables depends on the particular database used." ] }, { "cell_type": "markdown", "id": "3c2c59f6", "metadata": {}, "source": [ "An example of the complete set of all environment variables used by the GridAPPS-D platform is shown below" ] }, { "cell_type": "code", "execution_count": 6, "id": "26a1fe08", "metadata": {}, "outputs": [], "source": [ "import os\n", "os.environ['CIMG_CIM_PROFILE'] = 'cimhub_2023'\n", "os.environ['CIMG_URL'] = 'http://localhost:8889/bigdata/namespace/kb/sparql'\n", "os.environ['CIMG_DATABASE'] = 'powergridmodel'\n", "os.environ['CIMG_HOST'] = 'localhost'\n", "os.environ['CIMG_PORT'] = '61613'\n", "os.environ['CIMG_USERNAME'] = 'test_app_user'\n", "os.environ['CIMG_PASSWORD'] = '4Test'\n", "os.environ['CIMG_NAMESPACE'] = 'http://iec.ch/TC57/CIM100#'\n", "os.environ['CIMG_IEC61970_301'] = '8'\n", "os.environ['CIMG_USE_UNITS'] = 'False'" ] }, { "cell_type": "markdown", "id": "20e65385", "metadata": {}, "source": [ "The required and optional arguments for the `ConnectionParameters` class are described in detail in [Environment Variables](../03_databases/3_2_env_variables.ipynb)" ] }, { "cell_type": "markdown", "id": "435aa6c0", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "36445b93", "metadata": {}, "source": [ "## Connecting to the Data Source\n", "\n", "The next step is to connect to the database or file source that will be used for CIM model. A complete list of connection types currently supported are described in [Supported Databases](../03_databases/3_1_databases_overview.ipynb)" ] }, { "cell_type": "code", "execution_count": 7, "id": "60f50f57", "metadata": {}, "outputs": [], "source": [ "from cimgraph.databases.blazegraph import BlazegraphConnection\n", "database = BlazegraphConnection()" ] }, { "cell_type": "markdown", "id": "7d1b5e5d", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "2655ebc7", "metadata": {}, "source": [ "## Creating a Container and Graph Model\n", "\n", "CIMantic Graphs uses an EquipmentContainer class as the starting point for building a knowledge graph of the power system model. The supported classes are `BusBranchModel`, `NodeBreakerModel`, and `FeederModel`." ] }, { "cell_type": "code", "execution_count": 8, "id": "9222efed", "metadata": {}, "outputs": [], "source": [ "from cimgraph.models import FeederModel" ] }, { "cell_type": "markdown", "id": "8874aa74", "metadata": {}, "source": [ "The power system network is then created by passing the database connection, container object, and a boolean flag whether a centralized or distributed model should be built." ] }, { "cell_type": "code", "execution_count": 9, "id": "d3038906", "metadata": {}, "outputs": [], "source": [ "feeder = database.get_object(mRID=\"49AD8E07-3BF9-A4E2-CB8F-C3722F837B62\") #ieee 13 bus\n", "network = FeederModel(connection=database, container=feeder, distributed=False)" ] }, { "cell_type": "markdown", "id": "f9a085e4", "metadata": {}, "source": [ "The network is populated by default with all ConductingEquipment, ConnectivityNode, and Terminal class instances. The knowledge graph is indexed by the class type and then the device mRID.\n", "\n", "\n", "To view all instances of a particular class, use the `.pprint(cim.ClassName)` method. The default python print method and individual .pprint() statements can also be used on `network.graph` and individual objects.\n", "\n", "By default, only equipment classes and basic connectivity information (nodes and terminals) are loaded with just the UUIDs of each class to optimize memory usage." ] }, { "cell_type": "code", "execution_count": 10, "id": "5f5a1c4c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " {\n", " \"@id\": \"52de9189-20dc-4c73-bdee-e960fe1f9493\",\n", " \"@type\": \"Breaker\",\n", " \"Terminals\": [\n", " {\n", " \"@id\": \"1d81c7fe-e88f-41e3-a900-476ca6476ccd\",\n", " \"@type\": \"Terminal\"\n", " },\n", " {\n", " \"@id\": \"2847e06b-c8ed-41e6-b515-c61c9e8eb4b4\",\n", " \"@type\": \"Terminal\"\n", " }\n", " ]\n", " }\n", "]\n" ] } ], "source": [ "network.pprint(cim.Breaker) # View all instance data of type Breaker" ] }, { "cell_type": "markdown", "id": "329cca8f", "metadata": {}, "source": [ "```mermaid\n", "mindmap\n", " 52de9189((**Breaker**\n", " name: None))\n", " [\"Terminals\"]\n", " 1d81c7fe(**Terminal**\n", " name: None)\n", " 2847e06b(**Terminal**\n", " name: None)\n", "```" ] }, { "cell_type": "markdown", "id": "15ae92ce", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "90acff5d", "metadata": {}, "source": [ "## Automated Database Queries\n", "CIMantic Graphs contains automatic query generation routines for all classes and all supported databases using the `.get_all_edges(cim.ClassName)` method. This query obtains all attributes for all objects of that class type and expands the knowledge graph by one edge with default instances of all associated objects." ] }, { "cell_type": "code", "execution_count": 11, "id": "a13183df", "metadata": {}, "outputs": [], "source": [ "network.get_all_edges(cim.Breaker)\n", "network.get_all_edges(cim.Terminal)\n", "network.get_all_edges(cim.ConnectivityNode)" ] }, { "cell_type": "markdown", "id": "9696e9d2", "metadata": {}, "source": [ "```mermaid\n", "mindmap\n", " 52de9189((**Breaker**\n", " name: brkr1\n", " normalOpen: False\n", " open: False\n", " retained: True\n", " ratedCurrent: 400.0\n", " breakingCapacity: 400.0))\n", " [Location]\n", " 085bbe1f(**Location**\n", " name: None)\n", " [EquipmentContainer]\n", " 49AD8E07(**Feeder**\n", " name: None)\n", " [BaseVoltage]\n", " 2a158e0c(**BaseVoltage**\n", " name: None)\n", " [\"Terminals\"]\n", " 1d81c7fe(**Terminal**\n", " name: brkr1_T1\n", " sequenceNumber: 1)\n", " 2847e06b(**Terminal**\n", " name: brkr1_T2\n", " sequenceNumber: 2)\n", "```" ] }, { "cell_type": "markdown", "id": "3fc53077", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "5b5edc63", "metadata": {}, "source": [ "## Traversing the Knowledge Graph\n", "CIMantic Graphs associates CIM classes by creating direct references between in-memory object instances based on the naming and hierarchy of the underlying information model. " ] }, { "cell_type": "markdown", "id": "f7ca504a", "metadata": {}, "source": [ "To view the attributes of particular object instance, directly invoke the attribute from the UML class diagram." ] }, { "cell_type": "code", "execution_count": 12, "id": "491e5fb1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n" ] } ], "source": [ "from uuid import UUID\n", "breaker = network.graph[cim.Breaker][UUID(\"52de9189-20dc-4c73-bdee-e960fe1f9493\")]\n", "print(breaker.normalOpen)" ] }, { "cell_type": "markdown", "id": "bd732b34", "metadata": {}, "source": [ "To traverse the knowledge graph, no custom queries are required. Instead, directly invoke the UML association names that serves as references between objects in the graph:" ] }, { "cell_type": "code", "execution_count": 13, "id": "b94a9be6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "brkr1 connects nodes 650 brkr\n" ] } ], "source": [ "bus1_name = breaker.Terminals[0].ConnectivityNode.name\n", "bus2_name = breaker.Terminals[1].ConnectivityNode.name\n", "\n", "print(breaker.name,'connects nodes',bus1_name,bus2_name)" ] }, { "cell_type": "markdown", "id": "72909d53", "metadata": {}, "source": [ "```mermaid\n", "%%{init: {\"theme\":\"neutral\"}}%%\n", "flowchart LR\n", " 52de9189(\"**Breaker**\n", " name: brkr1\")\n", " 52de9189 -- Terminals[0] --> 1d81c7fe\n", " 1d81c7fe(\"**Terminal**\n", " name: brkr1_T1\")\n", " 1d81c7fe -- ConnectivityNode --> 7beddadd\n", " 7beddadd(\"**ConnectivityNode**\n", " name: 650\")\n", " 52de9189(\"**Breaker**\n", " name: brkr1\")\n", " 52de9189 -- Terminals[1] --> 2847e06b\n", " 2847e06b(\"**Terminal**\n", " name: brkr1_T2\")\n", " 2847e06b -- ConnectivityNode --> 94f822e0\n", " 94f822e0(\"**ConnectivityNode**\n", " name: brkr\")\n", "```" ] }, { "cell_type": "markdown", "id": "e8d9b0e9", "metadata": {}, "source": [ "No separate queries or mapping are required for measurment objects. Call the `.get_all_edges` method for each measurement class, and then obtain the measurements from the equipment object" ] }, { "cell_type": "code", "execution_count": 14, "id": "a0599da8", "metadata": {}, "outputs": [], "source": [ "network.get_all_edges(cim.Analog)\n", "network.get_all_edges(cim.Discrete)" ] }, { "cell_type": "code", "execution_count": 15, "id": "dd8c642d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name: Breaker_brkr1_A_1_C phase: PhaseCode.C bus: 650\n", "name: Breaker_brkr1_A_1_A phase: PhaseCode.A bus: 650\n", "name: Breaker_brkr1_A_1_B phase: PhaseCode.B bus: 650\n", "name: Breaker_brkr1_Pos_1_A phase: PhaseCode.A bus: 650\n", "name: Breaker_brkr1_Pos_1_B phase: PhaseCode.B bus: 650\n", "name: Breaker_brkr1_Pos_1_C phase: PhaseCode.C bus: 650\n" ] } ], "source": [ "for meas in breaker.Measurements:\n", " p = meas.phases\n", " print(\"name:\", meas.name, \" phase:\" , meas.phases, \" bus:\", meas.Terminal.ConnectivityNode.name)" ] }, { "cell_type": "markdown", "id": "4c52863d", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "fb374f41", "metadata": {}, "source": [ "## Utils Shortcuts\n", "\n", "CIM-Graph provides several shortcut methods using the utils library for file writing, simplified queries, diagram creation, and class visualization.\n", "\n", "The utils module can be imported using" ] }, { "cell_type": "code", "execution_count": 16, "id": "1047ec51", "metadata": {}, "outputs": [], "source": [ "from cimgraph import utils" ] }, { "cell_type": "markdown", "id": "6db2db56", "metadata": {}, "source": [ "### Shortcuts for Bulk Queries\n", "\n", "The utils module offers several shortcut methods for querying for all relevant classes involving lines, transformers, capacitors, or everything in the model. These methods are documented in [Utils Get All Data](../05_utils/5_2_get_all_data.ipynb). Due to the size of the queries (which are parsing tens of thousands of objects), completion can take a few seconds to a minute for large models.\n", "\n", "For example, the method to get all line data (including phasing, asset info, lower triangulare impedances, etc.) is" ] }, { "cell_type": "code", "execution_count": 17, "id": "31b0c6a2", "metadata": {}, "outputs": [], "source": [ "utils.get_all_line_data(network)" ] }, { "cell_type": "markdown", "id": "7c9d24c4", "metadata": {}, "source": [ "### Shortcuts for File Writers\n", "\n", "The utils module offers writers for XML, JSON-LD, and CSV formats for new and existing models. The arguments for the writers are the network GraphModel and filename / file directory:" ] }, { "cell_type": "code", "execution_count": 18, "id": "f891ac83", "metadata": {}, "outputs": [], "source": [ "utils.write_json_ld(network, 'my_model.json')" ] }, { "cell_type": "markdown", "id": "95a1f894", "metadata": {}, "source": [ "### Shortcuts for Visualization\n", "\n", "CIM-Graph offers automatic diagram creation for classes, object instances, and graph traversal paths using mermaid.js. The diagrams are plain text (enabling git tracking) and can be rendered in markdown and most web tools. Full documentation on use of the utils module to create diagrams is provided in [Mermaid Diagramming](../05_utils/5_3_mermaid.ipynb).\n", "\n", "When running in a Jupyter notebook, the `mermaid-python` library can be used to display the generated diagrams and can be imported using" ] }, { "cell_type": "code", "execution_count": 19, "id": "8c716161", "metadata": {}, "outputs": [], "source": [ "from mermaid import Mermaid" ] }, { "cell_type": "markdown", "id": "5ea3d95a", "metadata": {}, "source": [ "As an example of the diagramming capabilities offered, an arbitrary UML diagram can be create by passing in a list of CIM classes to be displayed. The diagrams presented above were also auto-generated using the utils module." ] }, { "cell_type": "code", "execution_count": 20, "id": "6e2e4b72", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", " \n", " " ], "text/plain": [ "