ManageIQ
Sprint 45 Review - Sprint End August 22, 2016
August 24, 2016
Overview
● Sprint Statistics (Oleg Barenboim)
● Providers (Greg Blomquist)
● Service Broker (Chris Kacerguis)
● User Interface (Dan Clarizio)
● Platform (Gregg Tanzillo)
● REST API (Alberto Bellotti)
● Automate (Greg McCullough)
● SmartState (Rich Oliveri)
● Performance (Dennis Metzger)
● Quality Engineering (Dave Johnson)
● Discussion
Sprint Statistics
(O. Barenboim)
375 Pull Requests Merged
PR Breakdown by Feature Category
(O. Barenboim)
* Note that some PRs
have more than one
category.
Providers
Sprint Comparison
(O. Barenboim)
All Repo Stats
(O. Barenboim)
Repository
manageiq 277
integration_tests 60
manageiq_docs 24
manageiq-ui-self_service 2
manageiq.org 8
azure-armrest 6
wrapanapi 4
handsoap 1
mangeiq-api-client 1
manageiq-api-client 1
manageiq-providers-amazon 4
guides 5
manageiq-appliance 4
pg-dsn_parser 3
ansible_tower_client 4
more_core_extensions 1
ovirt 6
azure-signature 6
linux_admin 3
manageiq-appliance-build 3
ovirt_metrics 1
Total 424
Providers
(G. Blomquist)
Providers - Hawkular (A.Bonas)
New functionality/features:
● Alerts
○ Link miq alerts and hawkular events on the provider
○ Convert miq alerts/profiles to hawkular group triggers/members of group
triggers
○ Sync the provider when miq alerts and alert profiles are created/updated
● 2 new entities added - domains and server groups
○ Including their visualization in topology
○ Limitation : still missing relevant icons that need to be designed by uxd
● Deletion operation of the datasource entity
● Support more event types for datasource and deployment
● Cross linking to VMs added to topology
● Add a button for Re-checking Authentication in the Middleware
providers list view
Providers - Hawkular (A.Bonas)
Misc
● Hawkular gem upgraded to 2.4
● Bugs fixed in breadcrumbs, alerts, live metrics, deployments and more areas
● Core UI - new Angular toolbars - will be mentioned in the UI section shortly
Providers - Google (G.Blomquist)
Preemptible Instances
Retirement support
Providers - Azure (G.Blomquist)
Additional Metrics
Fix proxy for Template lookups
Caching for SmartState
Providers - Containers (G.Blomquist)
Metrics status
Models for container deployments
Providers - OpenStack (G.Blomquist)
Host Aggregates
Region support
Providers - RHEV (G.Blomquist)
Snapshot Support
Providers - vSphere (G.Blomquist)
Block duplicate events
Storage Profiles
Providers - vCloud (G.Blomquist)
Video!
Providers - General (G.Blomquist)
Load Balancers
supports :smartstate_analysis
Service Broker
Service Designer
● Blueprint API is 90% done
○ edit and publish are still in development
● Continue to connect the service designer with the API
○ create, view, and delete working
Arbitration Profiles
● Arbitration profiles are a collection of pre-defined settings
● Works in conjunction with the Arbitration Engine
● Demo! (Jillian Tullo)
Service Broker
Rules Engine
● Will allow admins to easily create rule sets for to enforce policy
○ e.g. enforcing a VPC in AWS for development servers
● UX design is in progress
● API has been completed
User Interface
Dan Clarizio
● PRs merged (131)
○ SSUI (2)
○ Bugs (54)
○ Enhancements (26)
○ Tech Debt / Refactoring (57)
○ Backported to Darga (4)
● Refactoring
○ Converted to TreeBuilder - Manage Policies, Servers/Roles, SmartProxy
Affinity, Bottlenecks, Utilization, Compare Sections
○ Angular forms - Set Ownership, Time Profile Editor
○ Some simple RJS use cases replaced
○ Toolbar refactoring - Service Orchestration, VMs
○ Toolbar functionality accessible from Angular (Karel)
User Interface
● Enhancements
○ I18n support for UI plugins
○ Arbitration Profiles management for Service Broker
○ Re-check Authentication button added to Provider list views
○ Provisioning button added to the Templates & Images list and summary
screens
○ Subtype option added to Generic Catalog Items
○ About modal added to OPS UI
User Interface
Re-check authentication button on Provider list
User Interface
Provisioning button for Templates & Images
From list view:
From summary screen:
User Interface
Subtype option on Generic Catalog Item
User Interface
‘About’ modal in OPS UI
Platform
Gregg Tanzillo
Enhancements and Bug Fixes (90 PRs Merged)
● PostgreSQL High Availability
● Tenancy
● Additional Enhancements and Bug Fixes
● Rest API
Platform
Gregg Tanzillo
PostgreSQL High Availability
● Primary/Standby DB config in Appliance Console
● Database-only appliance config in Appliance Console
Platform
Gregg Tanzillo
PostgreSQL High Availability
● Failover Monitor
○ Runs as a service under systemd
○ Provides automatic failover of EVM server to new
primary DB
Platform
Gregg Tanzillo
Tenancy
● Work continues to support sharing resources across
tenants
○ Groundwork in preparation for supporting multiple
entitlements
○ ApplicationHelper#role_allows and User#role_allows?
combined and moved to Rbac
● Post refresh hook to queue mapping of Cloud Tenants
Additional Enhancements and Bug Fixes
● Upgrade ruby 2.2.5 to 2.3.1
● Configure Rails web server - Puma or Thin
○ Puma is still the default
○ Planning on adding additional servers
● DB maintenance scripts added to appliance
● Expression refactoring and cleanup
○ Relative dates and times, conversion to Ruby
Platform
Gregg Tanzillo
REST API
Alberto Bellotti
Continuing Refactoring
● 12 Refactoring PR’s merged
Bug Fixes
● Hide internal Tenant Groups from /api/groups
● Raise 403 Forbidden for deleting read-only groups
● API Request logging
REST API
Alberto Bellotti
Added support for BluePrints
● New collection /api/blueprints
● Supporting CRUD operations:
POST /api/blueprints to create
{
“name” : "bp_name",
“description” : “blueprint description",
“bundle” : {
“service_catalog” : { “id” : :service_catalog_id },
“service_dialog” : { “id” : :service_dialog_id },
“automate_entrypoints” : { "Provision" : "a/b/c", "Reconfigure" : "x/y/z” },
“service_templates” : [ { “id” : :st_id1 }, { “id” : :st_id2 } ]
}
}
POST /api/blueprints/:id action “edit” for single resource update
POST /api/blueprints action “edit” for bulk updates
DELETE /api/blueprints/:id for single resource delete
POST /api/blueprints action “delete” for bulk deletes
REST API
Alberto Bellotti
Added support for BluePrints
● GETs of blueprints include serialized content:
GET /api/blueprints/:id
{
“href” : “http://localhost:3000/api/blueprints/:id”,
“Id” : 123,
“description” : “blueprint description",
“content” : {
“id” : :st_id,
“description” : “blueprint description”,
...
“service_catalog” : { },
“service_dialog” : { },
“automate_entrypoints” : { "Provision" : "a/b/c", "Reconfigure" : "x/y/z” },
“service_templates” : [ ]
}
}
REST API
Alberto Bellotti
Added support for compressed id’s in inbound requests:
GET /api/vms/1000000000002
Can now also be requested as follows:
GET /api/vms/1r2
REST API
Chris Kacerguis
Added support for Arbitration Rules
● New collection /api/arbitration_rules
● Full CRUD support
POST /api/arbitration_rules
{
“name” : “admin rule”,
“operation” : “inject”,
“expression” : { “EQUAL” : { “field” : “User-userid”, “value” : “admin” } }
}
POST /api/arbitration_rules/:id action “edit” for single update
POST /api/arbitration_rules action “edit” for bulk updates
DELETE /api/arbitration_rules/:id for single Delete
POST /api/arbitration_rules action “delete” for bulk deletes
Automate
Greg McCullough
● Enhanced Messaging for Provisioning
● Old Message Examples:
Automate
Greg McCullough
● Enhanced Messaging for Provisioning
○ Displayed elements
i. ManageIQ Server name
ii. Name of VM/Service being provisioned
iii. Current Automate state machine step
iv. Status message
v. Provision Task Message
vi. Retry count (when applicable)
Automate
Greg McCullough
● Google Compute Engine - retirement support
● New tagging method - taggable?
○ Programmatically determine if a Service Model class or instance is
taggable.
● Generic Objects
○ Model Updates
■ Associations
■ Tagging
■ Service Methods: add_to_service / remove_from_service
○ In-progress: UI and Methods
Automate
Madhu Kanoor
● Git Automate support
○ Branch/Tag support
○ Contents are locked
i. Can be copied to other domains for editing
○ Editable properties
i. Enabled/Disabled
ii. Priority
iii. Removal of Domain
○ Dedicated server role to store the repository
● Future
○ REST API to refresh domains from a Git hook
Automate
Madhu Kanoor
● Git Automate - Demo
SmartState
Rich Oliveri
Deployed new MiqDiskCache module for use in Azure SSA
● Scan time:
○ From: Greater than 20 minutes
○ To: Less than 5 minutes (some cases < 2 minutes)
● Azure backend read requests per scan:
○ From: More than 7000 requests/scan
○ To: Less than 1000 requests/scan (some cases < 400)
Performance
Dennis Metzger
Focus - Page rendering performance
Page specific & Multi-page (common code)
Big Win (multi-page impact):
Test Page: Services -> Workload -> All VMs
Route: /vm_or_template/explorer
Database with 20,000 VMs
Before:93,770 ms
After: 524 ms
Quality Engineering
Dave Johnson
● Continuing to verify issues being fixed
○ Some 75 passed, 8 were reopened
● Began focusing on upstream nightly build for testing
○ Some 50 issues opened in the last 7 days
● Continuing to work on automation
○ Building more coverage around Middleware, Containers, Azure scenarios
○ Continuing to refactor for our next version of the fw
● We need a better solution around understanding when JS is complete in the
UI
● We continue to seeing a lot of string changes that affect the automation, we
need to get better about static element IDs
Discussion
Sprint 46 Review - September 14

Sprint 45 review

  • 1.
    ManageIQ Sprint 45 Review- Sprint End August 22, 2016 August 24, 2016
  • 2.
    Overview ● Sprint Statistics(Oleg Barenboim) ● Providers (Greg Blomquist) ● Service Broker (Chris Kacerguis) ● User Interface (Dan Clarizio) ● Platform (Gregg Tanzillo) ● REST API (Alberto Bellotti) ● Automate (Greg McCullough) ● SmartState (Rich Oliveri) ● Performance (Dennis Metzger) ● Quality Engineering (Dave Johnson) ● Discussion
  • 3.
  • 4.
    PR Breakdown byFeature Category (O. Barenboim) * Note that some PRs have more than one category. Providers
  • 5.
  • 6.
    All Repo Stats (O.Barenboim) Repository manageiq 277 integration_tests 60 manageiq_docs 24 manageiq-ui-self_service 2 manageiq.org 8 azure-armrest 6 wrapanapi 4 handsoap 1 mangeiq-api-client 1 manageiq-api-client 1 manageiq-providers-amazon 4 guides 5 manageiq-appliance 4 pg-dsn_parser 3 ansible_tower_client 4 more_core_extensions 1 ovirt 6 azure-signature 6 linux_admin 3 manageiq-appliance-build 3 ovirt_metrics 1 Total 424
  • 7.
  • 8.
    Providers - Hawkular(A.Bonas) New functionality/features: ● Alerts ○ Link miq alerts and hawkular events on the provider ○ Convert miq alerts/profiles to hawkular group triggers/members of group triggers ○ Sync the provider when miq alerts and alert profiles are created/updated ● 2 new entities added - domains and server groups ○ Including their visualization in topology ○ Limitation : still missing relevant icons that need to be designed by uxd ● Deletion operation of the datasource entity ● Support more event types for datasource and deployment ● Cross linking to VMs added to topology ● Add a button for Re-checking Authentication in the Middleware providers list view
  • 9.
    Providers - Hawkular(A.Bonas) Misc ● Hawkular gem upgraded to 2.4 ● Bugs fixed in breadcrumbs, alerts, live metrics, deployments and more areas ● Core UI - new Angular toolbars - will be mentioned in the UI section shortly
  • 10.
    Providers - Google(G.Blomquist) Preemptible Instances Retirement support
  • 11.
    Providers - Azure(G.Blomquist) Additional Metrics Fix proxy for Template lookups Caching for SmartState
  • 12.
    Providers - Containers(G.Blomquist) Metrics status Models for container deployments
  • 13.
    Providers - OpenStack(G.Blomquist) Host Aggregates Region support
  • 14.
    Providers - RHEV(G.Blomquist) Snapshot Support
  • 15.
    Providers - vSphere(G.Blomquist) Block duplicate events Storage Profiles
  • 16.
    Providers - vCloud(G.Blomquist) Video!
  • 17.
    Providers - General(G.Blomquist) Load Balancers supports :smartstate_analysis
  • 18.
    Service Broker Service Designer ●Blueprint API is 90% done ○ edit and publish are still in development ● Continue to connect the service designer with the API ○ create, view, and delete working Arbitration Profiles ● Arbitration profiles are a collection of pre-defined settings ● Works in conjunction with the Arbitration Engine ● Demo! (Jillian Tullo)
  • 19.
    Service Broker Rules Engine ●Will allow admins to easily create rule sets for to enforce policy ○ e.g. enforcing a VPC in AWS for development servers ● UX design is in progress ● API has been completed
  • 20.
    User Interface Dan Clarizio ●PRs merged (131) ○ SSUI (2) ○ Bugs (54) ○ Enhancements (26) ○ Tech Debt / Refactoring (57) ○ Backported to Darga (4) ● Refactoring ○ Converted to TreeBuilder - Manage Policies, Servers/Roles, SmartProxy Affinity, Bottlenecks, Utilization, Compare Sections ○ Angular forms - Set Ownership, Time Profile Editor ○ Some simple RJS use cases replaced ○ Toolbar refactoring - Service Orchestration, VMs ○ Toolbar functionality accessible from Angular (Karel)
  • 21.
    User Interface ● Enhancements ○I18n support for UI plugins ○ Arbitration Profiles management for Service Broker ○ Re-check Authentication button added to Provider list views ○ Provisioning button added to the Templates & Images list and summary screens ○ Subtype option added to Generic Catalog Items ○ About modal added to OPS UI
  • 22.
  • 23.
    User Interface Provisioning buttonfor Templates & Images From list view: From summary screen:
  • 24.
    User Interface Subtype optionon Generic Catalog Item
  • 25.
  • 26.
    Platform Gregg Tanzillo Enhancements andBug Fixes (90 PRs Merged) ● PostgreSQL High Availability ● Tenancy ● Additional Enhancements and Bug Fixes ● Rest API
  • 27.
    Platform Gregg Tanzillo PostgreSQL HighAvailability ● Primary/Standby DB config in Appliance Console ● Database-only appliance config in Appliance Console
  • 28.
    Platform Gregg Tanzillo PostgreSQL HighAvailability ● Failover Monitor ○ Runs as a service under systemd ○ Provides automatic failover of EVM server to new primary DB
  • 29.
    Platform Gregg Tanzillo Tenancy ● Workcontinues to support sharing resources across tenants ○ Groundwork in preparation for supporting multiple entitlements ○ ApplicationHelper#role_allows and User#role_allows? combined and moved to Rbac ● Post refresh hook to queue mapping of Cloud Tenants
  • 30.
    Additional Enhancements andBug Fixes ● Upgrade ruby 2.2.5 to 2.3.1 ● Configure Rails web server - Puma or Thin ○ Puma is still the default ○ Planning on adding additional servers ● DB maintenance scripts added to appliance ● Expression refactoring and cleanup ○ Relative dates and times, conversion to Ruby Platform Gregg Tanzillo
  • 31.
    REST API Alberto Bellotti ContinuingRefactoring ● 12 Refactoring PR’s merged Bug Fixes ● Hide internal Tenant Groups from /api/groups ● Raise 403 Forbidden for deleting read-only groups ● API Request logging
  • 32.
    REST API Alberto Bellotti Addedsupport for BluePrints ● New collection /api/blueprints ● Supporting CRUD operations: POST /api/blueprints to create { “name” : "bp_name", “description” : “blueprint description", “bundle” : { “service_catalog” : { “id” : :service_catalog_id }, “service_dialog” : { “id” : :service_dialog_id }, “automate_entrypoints” : { "Provision" : "a/b/c", "Reconfigure" : "x/y/z” }, “service_templates” : [ { “id” : :st_id1 }, { “id” : :st_id2 } ] } } POST /api/blueprints/:id action “edit” for single resource update POST /api/blueprints action “edit” for bulk updates DELETE /api/blueprints/:id for single resource delete POST /api/blueprints action “delete” for bulk deletes
  • 33.
    REST API Alberto Bellotti Addedsupport for BluePrints ● GETs of blueprints include serialized content: GET /api/blueprints/:id { “href” : “http://localhost:3000/api/blueprints/:id”, “Id” : 123, “description” : “blueprint description", “content” : { “id” : :st_id, “description” : “blueprint description”, ... “service_catalog” : { }, “service_dialog” : { }, “automate_entrypoints” : { "Provision" : "a/b/c", "Reconfigure" : "x/y/z” }, “service_templates” : [ ] } }
  • 34.
    REST API Alberto Bellotti Addedsupport for compressed id’s in inbound requests: GET /api/vms/1000000000002 Can now also be requested as follows: GET /api/vms/1r2
  • 35.
    REST API Chris Kacerguis Addedsupport for Arbitration Rules ● New collection /api/arbitration_rules ● Full CRUD support POST /api/arbitration_rules { “name” : “admin rule”, “operation” : “inject”, “expression” : { “EQUAL” : { “field” : “User-userid”, “value” : “admin” } } } POST /api/arbitration_rules/:id action “edit” for single update POST /api/arbitration_rules action “edit” for bulk updates DELETE /api/arbitration_rules/:id for single Delete POST /api/arbitration_rules action “delete” for bulk deletes
  • 36.
    Automate Greg McCullough ● EnhancedMessaging for Provisioning ● Old Message Examples:
  • 37.
    Automate Greg McCullough ● EnhancedMessaging for Provisioning ○ Displayed elements i. ManageIQ Server name ii. Name of VM/Service being provisioned iii. Current Automate state machine step iv. Status message v. Provision Task Message vi. Retry count (when applicable)
  • 38.
    Automate Greg McCullough ● GoogleCompute Engine - retirement support ● New tagging method - taggable? ○ Programmatically determine if a Service Model class or instance is taggable. ● Generic Objects ○ Model Updates ■ Associations ■ Tagging ■ Service Methods: add_to_service / remove_from_service ○ In-progress: UI and Methods
  • 39.
    Automate Madhu Kanoor ● GitAutomate support ○ Branch/Tag support ○ Contents are locked i. Can be copied to other domains for editing ○ Editable properties i. Enabled/Disabled ii. Priority iii. Removal of Domain ○ Dedicated server role to store the repository ● Future ○ REST API to refresh domains from a Git hook
  • 40.
  • 41.
    SmartState Rich Oliveri Deployed newMiqDiskCache module for use in Azure SSA ● Scan time: ○ From: Greater than 20 minutes ○ To: Less than 5 minutes (some cases < 2 minutes) ● Azure backend read requests per scan: ○ From: More than 7000 requests/scan ○ To: Less than 1000 requests/scan (some cases < 400)
  • 42.
    Performance Dennis Metzger Focus -Page rendering performance Page specific & Multi-page (common code) Big Win (multi-page impact): Test Page: Services -> Workload -> All VMs Route: /vm_or_template/explorer Database with 20,000 VMs Before:93,770 ms After: 524 ms
  • 43.
    Quality Engineering Dave Johnson ●Continuing to verify issues being fixed ○ Some 75 passed, 8 were reopened ● Began focusing on upstream nightly build for testing ○ Some 50 issues opened in the last 7 days ● Continuing to work on automation ○ Building more coverage around Middleware, Containers, Azure scenarios ○ Continuing to refactor for our next version of the fw ● We need a better solution around understanding when JS is complete in the UI ● We continue to seeing a lot of string changes that affect the automation, we need to get better about static element IDs
  • 44.